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ABSTRACT 



A record and playback system for video images, e^jecially 
suited for multi-camera industrial survciUance. Techniques 
for acquiring multiple asynchronous camera inputs, com- 
pressing video images, and storing digital image data are 
described. Selective resolution recording imp-ovcs object 
discernability without large large increase in data storage. A 
recording system with automatic data archive that eliminates 
the need for regular operator attention is disclosed. 
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1 2 

COMPRESSED DIGITAL VIDEO RELOAD switch closure or motion detected by infrared or video 

AND PLAYBACK SYSTEM motion detectors. The objective is to raise the probability of 

capturing signiftcant information at the time of ao event. 

FIELD OF THE INVENTION Many security managers elect not to use such techniques 
This invention relates to the recording and recovery of ^ since they disnipt the unifcHroity of media fill times and (he 

video and other data from single or multiple monitored sites. scheduling for dianging recording media. The same prob- 

It is particularly qjplicablc to surveillance of sites such as of non-uniform media fill rates is brougjil on by a 

industrial plants, banks, super-markets, and othe coiimicr- similar technique of recording extra images occurring priw 

dal and institutional properties. *c trigger event; tiie so-called *pre-trigger' recording 

10 techniques. I^-trigger recording, which has become prac- 

BACKGROUND tical with the advent of digital video methods, could be 
A primary value in video recording for surveillance and utilized more widely were the problem of media fill vari- 

security derives from gathering an accurate record of image ability solved. 

information from cameras and maintaining that record fcx* VHS videocassette recorders deliver horizontal resolution 

some duration of time, or archive interval (archive intervals for color images of approoumately 240 TV lines for standard 

of 24 hours to two months arc common). Conventional models and ^jproximately 400 TV lines fox S-VHS models, 

single-camera and multi-camera installations almost univer- Many videocameras available for security have horizontal 

saily use videotape recorders (VTR), primarily the VHS resolutionsof 330 or 460TV lines. Performance of either the 

format of videocassette recorders (VCR), to store tfie large camera or the VCR becomes wasted in the mismatch. Also, 

volume of camera images needed to record and replay ^ since resolution of VCRs is fixed by design, techniques to 

motion activity over useful archive intervals. raise or enhance the recorded image resolution in response 

Operation and maintenance of VCRs presents a consid- ^® particular monitoring conditions are not practiced in tiie 

crable cost and workload overhead in security systems, existing art of industrial surveillance, 

especially since most security systems are operated Several techniques have evolved in the field of security 

continuously, day in and day out. for the life of the instal- ^ and surveillance for adding other types of data into the video 

lation. VCRs generally require overhaul for every 10,000 stream for recording on the VCR. For exan^e. in U.S. Pal 

hours to 20,000 hours of operation (approximately 1-2 years No. 5,216,502 Katz describes how to record transaction data 

of continuous operation), and pictures can degrade as the from cashier lanes with the video images from the lanes by 

VCR accumulates hours of operation toward an overhaul impressing the digital transaction data onto the audio track 

date. of a VCR, or onto tiie end lines of the video frame recorded 

The relatively smaU number of images (432.000) that can on the V(3L In U.S. Pat. No. 4.^9,173 Mitsuhashi dis- 
be stored on a VHS cassette dictates that operators make cusses methods to carry continuous-time audio along with 
frequent cassette changes on an ongoing basis, or employ intermittently recorded (time-lapse) video images. Multi- 
many VCl^ recording at reduced rates so that cassette 35 P^^^^x devices (discussed further below) generally code 
dianges can be made according to a convenient schedule alarm status iriformation and camera source identification 
(e.g. once every 24 hours). Long term unattended recording ^^^^ video iiiput prior to storage on the VCR. 
by repeated cycling of a single tape is not advised since t^ Multiplexer devices are conunonly used in multi-camera 
life is on the order of 25 cycles, and no warning of image installations to choose single fields or flames from any of the 
degradation due to tape wear is provided on standard ^ asynchronous camera sources to create a video image 
machines. sequence for recording onto a VCR. Multiplexers are 

In short, VCRs arc bull^ relative to the number of images employed to maximize the rate of coverage at each camera 
stored and are prone to mechanical wear which causes view (to minimize the time b^een images) while mini- 
picture degradation. VCRs require maintenance far more mizing the total number of images that must be stored on the 
frequently than any other element of the surveillance sys- 45 recorder. Multiplexers also provide time-base correction for 
tem. It is acknowledged in the security industry that the VC!R continuous or time-lapse recording VCRs. which is neces- 
is the weak link in the task of recording massive amounts of sary since the rotating head-drum in a VCR cannot instantly 
image information on a routine y ear-in and year-out basis. synchronize to a randomly arriving field. Commercial mul- 

Pmposals for consumer VCRs and camcorders based on tiplexers for surveillance employ video analog-to-digital-to- 

digital rather than analog techniques are nearing accqjtancc 50 techniques and constitute a significant fraction of 

at the time of writing. A prx)posed Standard DefiniUon (SD) overaU system cost 

format for digital recording stores four and a half hours of FIO. 1 shows a representative conventional multi-camera 

NTSC equivalent video on a cassette that occupies 30% the system using a multiplexer 60. The time-multiplexed output 

volume of a VHS cassette. This represents a nearly eight- of die multiplexer is applied to a first VCR €2 f<x recording, 

fold in^>rovement in picture density over VHS technology. 55 Playback of previously recorded information from a second 

The mechanics of digital VCR recording, however, have VCi 64 is returned to a time-multiplexed input on the 

been chosen to link head rotational speed and tape speed to multiplexer. During playback the nmltiplexer selects one 

frame rate in order to easily inclement features such as camera view from the time-multiplexed playback signal and 

editing, still playback, and mulU- speed playback. This displays this camera view on a monitor 66. The multiplexer 

requires the compressed data stream to be of a fixed average 60 can also deliver live camera images to the monitor, 
length over intervals such as a frame or a fraction of a frame. 

This constraint limits the compression ratios that could be SUMMARY AND OBJECTS 

attained were a highly-variable-ratio con^)ression scheme A first object of the invention is to provide a long-running 

otherwise used To achieve greater picture density, an alter- video recording capability for surveillance applications that 

nate recording format will be required. 55 delivers equal or greater image storage capability than 

Many techniques have been devised to accelerate the rate conventional VCR techniques for lower overall equipment 

of image recording in response to trigger events such as door and operating costs. 
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Another object of the invcntioD is to provide an informa- 
tion record and recovery architecture that can be flexibly 
expanded to accommodate the many types of information 
arising in security and surveillance applications. These 
informations include motion detector inputs, alarm status, 
point-of-sale data, and audio. 

Another object of the invention is to provide a superior 
range of performance choices for video recording over that 
of standard VHS VCRs or proposed consumer digital VCRs. 
These include a choice of image resolution to match the 
performance of the camera; choice of image resolution to 
matd) tiie particular needs of a monitored view; and selec- 
tively higher resolution of critical portions of an image. 

Another object of the invention is to provide superior 
reliability and to lessen ca eliminate routine maintenance. 

Another object of the invention is to provide a single 
field/frame record and playback platform diat operates with 
a plurality of cameras without the use of an external mul- 
tiplexer device. 

Another object of the invention is to provide enhanccd- 
mode recording during an interval extending both prior to 
and subsequent to a trigger event Enhanced-mode recording 
constitutes an increase in Image rate, in image resolution, or 
both. 

Another object of the invention is to reduce the physical 
bulk of recording equipment in large site installations, and to 
reduce the number and size of media cartridges needed fos 
archival storage. 

Another object of the invention is to |M:ovide a true 
unattended recording system for surveillance that requires 
no operator intervention and operates reliably for the life of 
the equipment. This unattended recording system maintains 
a set of images recorded over the most recent archive 
interval for review or copy. This system fills a need fa- 
smaller installations where manpower is unavailable to cany 
out media changes, cleaning, or maintenance on a periodic 
basis. 

The invention digitizes selected flames or fields of video 
from a single video source or multiple asynchronous video 
sources, compresses the digitized video, and stores the 
compressed video onto a digital storage device. The inven- 
tion also provides means to decompress and play back the 
stored video images. Separate inputs on the invention accept 
alarm state data, audio data, ot other general alphanumeric 
or binary data. This additional data can easily be included 
into storage since the format of the storage device is inde- 
pendent of the data types recorded. The invention is basi- 
cally a flexible data storage system which can accommodate 
new data sources without the need to convert and fit data to 
a preexisting format as is required for dedicated video 
recorders. 

FIG. 2 shows a general multi-camera record and playback 
application according to principles of the invention. A video 
selector 70 between camera inputs and the digitizer selects 
single or multiple fields or flames from cameras. By tracking 
the phase of each camera signal as it is digitized, the order 
of camera selection can be arranged such that single fields or 
flames can be acquired at neariy the full rate of the video 
signal format This feature enables the invention to replace 
the multiplexer normally used ahead of the VCR. resulting 
in significant cost savings. 

Video fields or flames selected by the selector pass to a 
video digitizer 72 and a image confessor 74. The digitized 
images are compressed by algorithms designed to operate 
specifically with fixed-view, low motion content images of 
the type characteristic of surveillance applications. These 
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algorithms, which produce a variable data rate as a function 
of image content achieve higher average compression ratios 
than do compression techniques designed for general video 
program content It is postulated that average compression 
5 ratios of 100: 1 or greater can be adiieved in typical surveil- 
lance applications. At 100:1 compression a 4&0x360 pel 
8-bit monodirome image is reduced to less than 2 kilobytes. 
High compression ratios enable smaller storage devices, a 
key to enabling reliable, cost-eflFectivc alternatives to VCRs. 
10 The compressed data rate associated with each camera 
view in a monitored site varies widely as a function of image 
content and motion. Tliis content and motion varies from 
camera view to camera view and varies also according to 
time of day. The invention achieves low average confessed 
IS data rates across multiple camera views by exploiting the 
averaging effects of combining a large number of com- 
pressed data streams over the time interval available foe 
filling stcKBge media. Multi-source averaging causes data to 
accumulate in storage at a more uniform rate than would be 
^ the case If data was accumulated firom a single camera 
source alone. Acctmuilation of data over long time intervals, 
e.g. 24 hours, also lessens the effect of rate variations, 
caused by daily activity cycles, on media fill times. 
A rate control system is incorporated in the invention to 
^ assure consistency in storage c^adty fill times in the 
presence of varying compressed data rates. Hie rate control 
feature changes the rates and resolution at which images are 
captured in orda to fill storage capacity at or nearly within, 
a target time interval. An additional rate control feature 
^ anticipates daily and weekly data rate patterns to achieve fill 
time targets while maintaining a high degree of regularity in 
image rates and resolution. 

The invention described affords wide latitude In choosing 
the rates and resolution of recorded images. The resolution 
of existing art VCRs is essentially fixed by design, with two 
resolutions available in VHS format — standard and 
coII^)oncnt-video (S- Video). In contrast, the resolution of 
each image retained by the invention can be chosen inde- 
pendently to match the performance of cameras and to 
^ respond to the requirements of scene content and event 
triggers. Equipment can be designed to deliver resolution 
greater than that of standard and con^nent-video VHS 
recorders. 

^5 The invention can save images that occurred prior to a 
trigger event by temporarily storing images from selected 
cameras at rates greater than the normal record rate. If no 
trigger arrives before an expiration time the temporary 
images are discarded. If a trigger arrives in time however the 
5Q temporary images are re-assigned to a mass storage device. 
Saving pre-trigger images in this fashion can provide a more 
comprehensive record of an event. 

A first embodiment of the invention stores the compressed 
images on a removable-media storage device 76. such as a 
35 magnetic digital tape drive. This arrangement is cost effec- 
tive when it is desired to archive large mounts of data or 
when data must be retained for a long archive interval. Rate 
control techniques cause media cartridges to be filled in 
specified target intervals even in the presence of variable 
^ compressed data rates. 

A second embodiment of the invention uses a fixed-media 
storage device 78. for exan^le a magnetic disk drive. This 
arrangement permits completely automatic, unattended 
recording for indefinite periods of time. No operator is 
65 needed to routinely change disk or tape cartridges. A data 
replacement process overwrites the oldest images on the 
storage drive. Rate control insures that new data is written 
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at a rate such that a Tna-gimum number of images are stored coDventional VHS format tape (432.0CX) fields per VHS 

without losing data newer than the archive interval. t^), or 2.5 times as many images as will be carried on the 

FIGS. 3A and 3B show how image data is stored and proposed SD fonnat cassette for digital VCR. As relations 1 
retained for the duradon of an archive interval in removable- through 3 suggest, this additional storage edacity can be 
media and fixed-media embodiments of the invention. Car- 5 traded for higher image recording rates, higher image 
tridges are the basic unit of storage in the removable-media rcsoUitions, or longer recording times, 
embodiment. The archive storage loop is implemented by Equipment cost advantages: It is postuUted that equip- 
frfiysically cycling media cartridges through an external ^^^^ can be produced by use of the invention that stores an 
caitiidge shelf. By contrast, the archive stwagc loop in the greater number of images per doUar of equipment 
fixed-media embodiment shown in no. 3B is internal to the lo cost than by conventional VO^ technology. This result 
mass storage device. A series of data records is retained as becomes possible as the cost of digital stOTage and the cost 
archive within the storage device. A data replacement pro- of the hardware to execute comi»:ession deaease. Fixed- 
cess aUows data records to be discarded as they ages past the view camera images of the type found in surveillance 
ardiive interval making room to write new data records. The appUcations can be corajH-esscd mwe deeply than general 
fixed-media embodiment of the invention therefore auto- 15 movie or broadcast program content Average compression 
mates the process of recording, retaining and discarding ratios of 100:1 or greater in surveillance applications are 
(jata expected. By contrast, VCRs stOTe analog data in non- 

MaintainabiUty advantages: The removable-media compressed form. The ft-ndai^ntal advantage of image 

embodiment using a digital ta^ drive writes several times as compression represents a large offsetting factcn- against other 

many images to tSpe jir mintite of tape drive head rotation co^s inherent m the approach described by the invention. 

« / t \;r^. TTUj. ^^^a a^«o«*onr- !o « Furtfier cost advantages accrue from replacement of the 

as do analog VCRs. This wntmg speed advantage is a , . , . _^ ^ • 

* , . • J J „ *u * multiplexer in present-art security systems. The invention 

consequence of image oon,press,on and reduces tt« rate of ^ ^ function^p.rfarmed by the multi- 

head and mechanical dnve wear per number of images , * ^ ,u . , , r , a ^ 

recorded. Maintenance intervals for Ae equ^ment are con- ^'^ *^ "'^'^^ » ""''S video 

digiu. upe .n^. |£invenUon orated wjj a 

tape drive transferring 500 toytes/second stores two- ^^^^^ ""^ ^ "i i 

hundred fifty 2 kbyte ^ges per second. The digital tape f^"^^^ "^^^ ""^J^J^ '^"^"^^ 

embodin^thus h^ an ilge^ting speed advantageVf ^ ^^"f machines as frequenUy saving maopowe^. 

50:1 relative to a 24-hour Le-lapse VCT- Assurig a ^^f^ ^^^'f;^ rtS^HiT™^! 

similar mean time between overha!a for the two types of ^ « l^'^'^J^cT 

drives then the digital tape embodiment would be eijLted ^ ^o^^l ^ ^HS tapes while holding five times as 

to write a far greier wL^ of images between ov^auls "nagcs^This volumctnc advantage of 25:1 results in 

than the analog VCR. ^ymg scheme requiring less storage space-a 

"6 ^ significant gain in large mstallations. 

The fixed-media embodiment can operate a sealed unit .^^^^^ features and advantages of the invention 
disk drive witti heads that do not contact the disk surface. ^ understood in view of the accompanying 
Disk drives have dcmonstiatably better lifetime than t^ drawings and the following detailed description. The draw- 
drives when operated under conditions of continuous data ^ ^^^^^^ exemplary embodiments of the invention and 
transfer. illustrate various objects and features thereof. Use of func- 

Perfomaance advantages: The relationship below holds ^Jqhs ^nd systems appearing in the drawings and description 

for both conventional VTRs and compressed-image record- are Dot limited to the context of the exemplary embodiments, 
ing systems of the type represented by the invention: 

45 BRIEF DESCRIPnON OF DRAWINGS 

data stDr^ge=image reccnding latexdata per 

iaugexrocording time (relation I) piQ^ 1 is a block diagram of a prior-an multi-camera 

where in a non-compressed video recording system: recoTiling and playback appUcation. 

FIG. 2 is a block diagram of a general multi-camera 

data per imag&»<image resolution (relatioD 2) application using the invention. 

and in a compressed video recording system: FIG. 3A shows the data ardiive cycle for a conceptual 

removable-media recording application. 

^^^^^ ^j.^. image resolutkm (relation 3) pjQ 35 shows the data ardiive cycle for a conceptual 

compression ratw fixed-media recording appUcation, 

Compression ratio is a function of both the amount of 55 piG. 4 is a block diagram of an in:q)lementation of the 

detail in a single image and ^e change of detail, or motion. record function of the invention. 

occurring between images. PIQ 5 jUustiates data flows for an implementation of the 

The relations suggest that for a given data storage ci^c- record function 

ity the factors of recording rate image rejolwiom and pjo^ , y,^^ ^ ^ ^„ implementation of the 

recording time, or time to fill avaUable storage capacity, can 60 , , , ^. , ^ . ^ ^ 

betradedoffagainsteachotheraccordingtothencedsofa playback ftincUon of the mvendon. 

particular application. i^ustrates data flows for an implementation of the 

A digital tape embodiment of the invention can store 2.5 playback function, 

million *VHS equivalent' images (360x240 pels) on one 8 FIG. 8 is a detail block diagram for source selection of 

nun tape assuming a 5 gigabyte tape drive and further 65 asynchronous video inputs. 

assuming two kilobytes of data per con^ssed image. This FIG. 9A is an example of selecting fields from asynduo- 

is more than five times as many images as are stored on a nous sources where the source sequence is arbitrary. 
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FIG. 9B is an example of selecting flames from asyn- FIG. 32A shows an example of an image resented with 

cfaronous sources where the source sequence is arbitrary. 320x240 picture elements (pels), 

FIG. 9C is an example of selecting fields from asyachro- FIG. 32B shows the image of HG. 32A represented with 

nous sources where the source nearest in phase is chosen 640x480 picture elements (pels). 

next ^ FIG. 32C shows a split-resolution rq)resentation of the 

FIG. 10 is an example of switching between video chan- image of FIG. 32A. 

ncls of different phase in the vicinity of the vertical syn- pjQ ^2D shows an expansion of the windowed portion of 

chronizing intcrvaL FIG. 32C. 

FIG. IIA is a block diagram of a paior-art interframe pjQ 33^ difference, or delta, of two images 

image compressor. where the delay time between images is small. 

FIG. IIB iUustrates the date flow topology for the inter- pjQ 33^ ^^iow^ the difference, or delU, of two images 

frame image confessor of FIG. IIA. where the delay time between images is large. 

FIG. lie is a block diagram of an interframe image 3^ ^^^^ ^ method for detamining the location of 

compressor for multi^e streams of images. 15 ^ moved object by evaluating three source images. 

FIG. IID is a block diagram of a pipelined version of the 35^ ^^^^ ^ ^^.^j. ^ recording system with j^e- 

interframe image corajMessor of FIG. IIC. recording capability. 

FIQ. 12A is a flowdiart of Rq)lacemcnt Process A used ^ ^^^^^ ^ recording system witfi pffe-trigger 

for overwriting stwage applicaUons. recording cqwbiUty according to the invention. 

FIG. 12B is a flowchart of Replacement Process B used 20 ^ ^ ^^^^^ ^^^^ 

for overwriting storage applications. prc-trigger and post-tiiggcx recording system of FIG. 35. 

FIG 13 is a block dia^ of fundamental rate control ^^^^ ^ applications example of a fixed-media 

appUed to a recording system. rcoord4>layback device, named an ^unattended recording 

HG. 14 is a block diagram of a recording system f<x ^ monitor' 

digital video inputs with rate control, also /ncorp^ating 38 is a detail block diagram for source selecdon of 

parameters of compression in the rate conOrol process. asynchronous digital video inp^T 

FIG. 15 is a block diagram of a recording system for -r 

analog video inputs with rate control. DEFINmONS 

J^^^nil' ^T"^ \ ^^^^ 30 ^ ^^^j jhe words 'image\ 'pictureV 

being ftUed by a data stream of variable rate. ^frame\ and Mdeo^ are used with the foUowing understand- 

FIG. 17 is a time history exan^le similar to HG. 16 where .^^j. ^^d ^picture' are used interchange ably and 

a rate modifier has lowered the storage data rate. ^^^^ representation of visual information either in 

FIG. 18 is a block diagram of a rate control system digital or analog form. 'Video' implies a scries of images 

according to the principles described for basic control loops. 35 (^^nected in time sequence to convey motion or timc- 

FIG. 19 is a time history example illustrating the response related activity. *Video' also in^^lies analog video signal 

of the basic control loop to step changes in base data rates. formats such as NTSC, PAL. and others « of which fields and 

FIG. 20 is a time history exanq)le illustrating the response frames are the disaete elements. A field or frame also 

of the basic control loop to varying base data rates. represents an itnage. In common usage, *frame* can refer to 

FIG. 21 is a 7-day time history example of base data rates ^ a single image out of a scries of images, 

in a weekday-staffed work site. The invention disclosed herein is capaWe of storing 

FIG. 22 is a block diagram of a recording system witfi rate pictures and images from video sources and replaying them 

control, including rate profile functions fcH" implementing in time sequence and at a rate necessary to convey the effects 

enhanced control loops. 45 of action and motion. The invention can be operated at 

FIG. 23 is a time history exanmlc of unsealed profile data reduced image rates as is commonly practiced in suryeil- 

lance and security WOTk to extend time coverage for a given 

TTwr^ u *u fil A^*^ «f vrn '>'\ c^oi^ storage capability. The invention can also store and regcn- 

FIG. 24 shows the profile dau rates of HG. 23 scaled by ^^^^^ ^j^eo sources at the full rate 

a jffofile rate modifier. ^^^^ ^ ^^^^ standards such as PAL and NTSC. For 

FIG. 25 is a block diagram of a rate control system so t^ese reasons, the invention is considered as a Wideo recoid- 

according to the principles described for enhanced control , device*, 
loops. 

FIG. 26 is a time history example that conqjares the twse DETAILED DESOaPTION OF THE 

data rate for a new recording cycle with a profile rate derived INVENTION 

from earlier cycles. Pjq 4 is a block diagram showing an implementation of 

HG. 27 is a time history example illustrating the response funcUon of the invention. FIG. 6 similarly shows 

of the enhanced control loop to the base data rate of HG. 26 hardware blocks for the playback function of the invention, 

while using the profiles of FIGS. 24. ^ s^>arate record-only and playback-only instruments 

FIG, 28 is a lime history ci base data rate to be used in the ^ can be constructed in accordance with the removable-media 

examples of FIG. 29 through FIG. 31. storage embodiment of the invention. Alternately, combined 

FIG. 29 illustrates the response of the basic control loop record/playback instruments can be constructed for both the 

for overwriting storage. removable-media and the fixed-media storage embodiments 

FIG. 30 illustrates the profile rates derived from the first by combining the record and playback functions and climi- 

seven days of the daU of FIG. 28. 65 mating redundant blocks. 

FIG. 31 illustrates the response of a composite basic/ Fot simplicity of illustration the recced function will be 

enhanced control loop for overwriting storage. described with the example of eight camera sources con- 
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nected as inputs. In practice, the record function could t>e to a computer graphics monitor 144. A wide variety of 

devised to accept any number of video inputs, including a graphics adapters and graphics mooiton are commercially 

single camera input or a timc-raultiplcxcd input containing available for personal computers (PC) and workstations. A 

multi-source video. second means connects a video converter 142 to the pro- 

A video selector 70 selects one of the eight camera signals 5 cessOT bus. The video converter generates composite or 

and routes this signal to an output terminal 100, The selector conqxment-video for viewing on a standard TV monitor 66 

output terminal is coupled to a video digitizer 72. The video or fw transfer to a VCR 62. 

digitizer extracts a digital representation of the image data An audio converter 146 reproduces audio files via an 

and the synchronization signals from the analog video. The audio transducer 148. Remaining hardware blocks shown in 

video digitizer may also be capable of detecting codes that lo FIG. 6 include a CPU. processor memory, and means for 

have been inserted in the vertical synchronizing interval of user and remote interface as appeared also in the recOTd 

the video source input. An input section for con^nent function described in FIG. 4. 

(Y/C) video (not shown) can also be devised with parallel Operation 

selector paths and digitizers for the Y and C signal compo- The record function wlU be described with reference to 

nents. 15 the record hardware blocks shown in FIG. 4 and the record 

An acquisition controUcr 104 controls the state of the diagram of HG. 5. For simpUcity. the following 

selector which couples video signals to Ac digitizer by way discussion assumes the example of recording NTSC video 

of selector control 114. Synchronization data 116 is received fields per second). 

by the acquisition controller from the digitizer. The acqui- Cam^a sources in surveillance installations, if not dnven 

sition controUer utilizes this synchronization data to detect ^ external synchronizing signals, can be considered as 

the start and end of fields, the phase of fields relative to a asynchronous, meaning image fields do not begin at iden- 

system clock 106. and the type of fields— odd. even. etc. The Cameras are sometimes locked to AC power line 

acquisition controller also generates a field/flame-valid 119 frequency. In such a case, field periods (Inverse of field rate 

signal under the conditions that an entire portion of a field ^r frequency) wiU match between cameras but field phases 

containing image dau was digitized during the time a single " Cameras may alternately run on mdependcnt 

video source was selected. The acquisition contioUer com- timebases. Field periods may then differ slighUy, causing 

municaies with a CPU 122 via a processor bus 112. ^^Id phases to continuaUy advance or regress relative to 

A digital field/frame capture 102 receives image data and , _^ „ . _ , * > 

synchr^Dizatlon dau fro^ the video dx&U^Thc field/ ^ ^ ^h* Yj^f" T f"!"?"^*" 

frame capture stores an a«ay of Image <Uta from a single » f ^Jf the mveaUoo to acqmre single video 

video field or a pair of fields ?eprese2.g a frame. SynchJo- f^<^ « frames fi^ nmltiple asynchronous sources accord- 

nization data 116 is used to ke^the iw^c daU into correct "8 to two switehing altemaUves^ A first switching alterna- 

airay locations. Capture of a field or TL commences on « ^ ^^'^ 

recent of a start fiddrtiame 118 signal from the acquisition *.1«»«=* '^fflf "^i? ''^'^ ** 

cond^ller. When the fleld^rame ^mre receives a field/ " video input rate (eg. 30 fidds per second). These rate 

frame-vaUd signal the image dau retained in the array is »«n,Ut.ons are miposed by tte flindamenuls of selectmg 

designated a complete iWe. The image dau is then ttins- asynchronous video signals by way of a smgle selector A 

f^Zi .h. j^IT- -JA second switchmg alternative offers an increase m single- 

tetred to tne image conmressw 74. _ , . . . i_ n . .u • ^ . \L < 

. , „ ... Held acquisition rates by allowing the mvention to control 

Data frona non-image sources including audio, alarm ^ ^^^^ s^\^on according to relative 

states, and otiier alphanumeric data are cwipled to a penph- ^^^^ ^^^^^ Acquisition at minimum average rates 

eral data interface 120. Inputs from^^ of^/^, of the inputrateforNvidcosourcescan beachieved 

appUed to an audio digitizer 130, the output of which ^^'^^ technique. The average acquisition rate for sixteen 

couples to the peripheral data interface. asynchronous camera sources, for example, would become 

Several other devices connect to the processes bus 112. 45 5547 f^^x^ ^ second. Of course, sustained acquisition of 

These include one ot more fixed-media storage devices 78 multiple fields or frames from single sources is possible with 

(e.g. magnetic hard-disk drive), one or more removable- either switching alternative at the normal fiUl rates of 60 

media storage devices 76 (e.g. magnetic digital tape drive, f^^^ ^ stconti or 30 frames per second, 

rewritable optical disk drive, or floH>y disk-drive), central- pi^^ g represents a portion of the hardware shown in FIG. 

processor (CPU) 122 and processor memory 124, means for 50 4 in greater detail. Video sources A, B, C. and D. jroduces 

implementing a user interface 126, and means for coupling streams 220A-D which are coupled to video selector 

to remote interfaces 128. 70. An ouq)ut video stream appears at die output 100 of the 

FIG. 6 shows hardware blocks for the playback function selects. Acquisition controller 104 operates the selector by 

of the invention. All hardware blocks connect to a common way of selector control 114. 

processor bus 150. Removable-media storage drive 76 55 FIG. 9A shows an example of how whole single fields are 

accepts media recorded on a separate recording device. A selected from multiple asynchronous sources according to 

fixed-media storage drive 78 is shared with the record the first switdiing alternative. This example assumes the 

function in a common record/playback device. Removable- four asynchronous sources and selector shown in FIG. 8. 

media transfer drive 80 permits re-recording of image Any number of video sources could be selected in practice, 

sequences from the primary storage devices as will be 60 A source switching sequence of A-B-C-D-A is used in this 

described later. The implementation shown assumes image example. Referring to FIG. 9A. the selector first selects 

decompression for playback will be in^lemented in soft- video stream A to direct field 1 to the selector output 

ware running on the CTU. Dedicated hardware for decom- Immediately after the portion of field 1 containing image 

pression are also considered under the invention. data has terminated the acquisition controller directs the 

IWo methods can be chosen for displaying playback 65 selectcr to select video stream B. A partial image field from 

images and text as indicated in FIG. 6. A computer graphics video stream B passes before a whole field, which will 

adaptor 140 connects to the processor bus to supply images constitute field 2 in the output stream, begins. Switching 
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condDUCS after field 2 is complete to select whole fields 3, horizontal lines. But channels 3 and 4, which match and lead 
4. and 5 from video streams C, D. and A respectively, with the phase of diannel 1 respectively, could also be selected 
partial fields occuiring in between. Note that partial fields since serration or equation pulses remain in these channels 
are always less than one field interval in duration. Since for measuring phase. Channels 1 and 3 represent the case 
partial fields are never longer than the desired whole fields, 3 where video signals arc exactly synchronous. Channel 4 
the net rate of whole fields appearing in the ou^ut stream represents approximately the laigest phase lead (+6 hcri- 
always equals or exceeds one-haft the source field rate. zontal lines) that still allows a phase measurement to be 

If a frame is c<Mi^sed of two contiguous fields and the made during the vertical syndironizing interval, 
field order is unin^wrtant (odd-even or even-odd) then a As each stream is digitized its phase is measured and 
maximum average of 20 such flames per second can be lO updated in the corresponding phase register 222A-D. 
produced in the output stream with the first switching Updating strcamphase every cycle corrects for source phase 
alternative (up to one field will be lost far each two fields jitter and even for progressive phase cnor caused by small 
acquired as a frame). An cxaiiq)le of this is presented in FIG. differences in the fidd period of the video sources. As a 
9B. If a certain field order witfiin a frame— odd/even or result the source order sequence may vary from cycle to 
even/odd— must be preserved then maxinmm average rate 13 cycle, A margin of phase error can be tolerated in source 
drops to 15 frames per second, or one-haft the source rate. selection as illustrated in the difference between switching 

The field rate of the ouQ)ut stream can be increased by channels 1 and 3 and switching channels 1 and 4 in FIG. 10. 
using the second switching alternative which drives the The phase difference between channels 3 and 4 is six 
order of source selection according to the field phase of the horizontal lines. Splitting this phase as ±3 lines, and dividing 
input video streams 220A-D. This alternative is tolerable fOT 20 over one field interval of 266 lines in^lics that a phase error 
surveillance monitoring since the frequency of camera views of ±3 divided by 266, or ±1,1%, can be tolerated in csti- 
is important— the recording sequence of cameras is not mating the phase of video sources, 
generally parent to the viewer. With this alternative field The phase error tolerance provides some measure of 
and ft^e rates in the output stream approach that of the tolerance against field period mismatdi, though not against 
input streams when the number of cameras is large. The 25 a mismatch which would cause more than ±1.1% phase 
example of the second switching aUernaUve shown in FIG, change between phase updates. Constant magnitude field 
9C begins with field 1 being selected from video stream A, period errors cause progressive accumulation of field phase 
as was done in the cxan^le of FIG. 9A. The next stream error. A much greater t(^crance against field period errors 
trailing closest in phase to video stream A is video stream D. can be achieved by implementing a second set of registers 
By selecting stream D after stream A the smallest partial 30 (not shown) to track the field period error of each video 
field, or unusable interval, is lost. Following stream D the source. Fidd period error is determined by measuring how 
selector connects in turn stream C, stream B, and then much field phase changes in the interval between field phase 
completes the cycle by returning to stream A. A total of one samples. 

full field interval was lost during the cycle. Hence, the The object of using field phase and field period error 
maximum average rate is times titie source stream rate 35 registers is to estimate the phase of a video stream priw to 
as cited earlier, where N is the number of video sources. selecting that same stream for digitization. Field phase at 
In a practical application of the second switching any time, t, is estimated as the phase last measured at time 
alternative, the acquisition controUcr 104 maintains jrfiase zero plus the field period error multiplied by time, L Of 
registers 222A-D to store a digital phase value for each course, apparatus can be devised to measure the phase of 
video stream (sec FIG. 8). These phase registers may be 40 input streams independenUy of tiie video selector. This 
implemented by software running on the CPU. Riase reg- obviates the need f<x phase and period register values, 
ister A 222A, for example, stores the most recently measured though it somewhat raises the cost of hardware, 
phase value for video stream A 220A. Phase measurements Returning again to HG. 4, the CPU directs acquisition of 
are derived from synchronization data 116. This phase data fields or flames via the acquisition controller 104. It is 
is contained in the video stream at the ouq)ut of the video 43 inched, but not shown in the figures, that the CPU can alter 
selector 70 and is ^pUed to the acquisition ccmtroUer. On the size of the capture array by changing the video digitizer 
start-up of the recording instrument the acquisition control- sampling rate to acquire more or fewer picture elements. 
Icr first determines the phase of all video input streams Capturing two successive fields as a frame doubles the 
220A-D by operating the selector such that the phase of vertical dinaension of the array size. Either a field or a frame 
each stream can be measured. Initial phase values are 50 will be treated as a single image by functions subsequent to 
established for eadi source relative to the system clock 106. the field/frame cq>ture blodc 

Following start-up the recoffder begins an image acquisition The image compressor 74 receives images from the 
cycle on an arbitrary input stream. It then proceeds by field/frame capture 102. The image compressor can perform 
progressively selecting streams which offer minimum delay cither intraframc or interfrarac compression of images, 
to the start of image data relative to the stream just com- 55 Intraframc compression generates a set of data fOT each 
pleted This process continues until images have been original image which, taken alone, sufficiently represents the 
acquired from all requested streams, after which the aoqui- original image. Intraframc compression is often referred to 
sition cycle can begin again. The phase value associated as stUl-imagc compression, as still images are treated inde- 
with each stream is updated in the corresponding phase pendently without relation to other images in the time 
register as each stream is acquired. 60 sequence. Ittterframc compression by contrast intends to 

FIG. 10 illustrates possible phase relationsh4)s at the time generate a data set which is smaller than that generated by 
video sources are switched. The figure hypothesizes four intraframc compression for the same set of images. Inter- 
different channels, each containing a video signal near the frame compression exploits the similarities, or 
time of the vertical synchronizing intervaL Switching from redundancies, within a set of images to reduce the amount of 
channel 1 to any of channels 2. 3. or 4 is assumed. Switching 65 data needed to represent the set. 

occurs after line number 525 on channel 1. Channel 2 is a Methods based on transform coding such as DCT 
possible next source since it lags channel 1 by only two (discrete-cosine transform) or wavelet transforms are suii- 
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able for coni^essing surveillance video. JPEG Is a DCT- 
based intraframe technique for still images. MPEG is a 
DCT-based interfirame technique for moving images. A 
tremendous cost advantage may materialize for MPEG 
compressors as VLSI implementations develop and mature. 
However, VLSI implementations for MPEG may lack tfie 
flexibility to be optimized for surveillance video recording 
q>plications. Also, the computationally intensive motion 
estimation processes inq}lementcd by MPEG integrated cir- 
cuits may be of limited value foi the fixed-view image 
sequences characteristic of surveillance video. 

Surveillance images can be expected to have a high 
degree of ten^x>ral redundancy since image sets are obtained 
from a single camera without transition to other video 
sources. Frequently a camera is physically fixed and any 
moving objects within the camera field of view occupy only 
a small fraction of the field of view. A high degree of 
con^jxession can be obtained with the simple interframe 
technique of encoding a first image as a key image and then 
encoding the remaining images as deltas taken relative to ttit 
preceding image. Encoding can be done by DCT, wavelet or 
other suitable tedinique. Coding errors are canceled out by 
including the encoder along with a decoder identical to the 
one used for decompression/playback within the loop that 
generates the delta image. This prior art technique is illus- 
trated in FIGS. UA and IIB. 

In FIG. IIA. unconq)ressed input images are applied to 
the non-inverting input of difference element 200, Selector 
A 210 supplies data to the inverting input of the difference 
element and to a summing element 206. The output of tiie 
difference element couples to encoder 202. Encoder ouQJut 
is delivered from the compression function as cwnpressed 
output Images. Decoder 204, which periorms the inverse 
operation of encoder 202, receives compressed images from 
the encoder output and delivers decoded images to the 
summing element The output of die simmiing element is 
delivered to an image store 208. Image store output is 
applied to one input of selector A. 

If a key image is to be generated then selector A selects 
a zero data value representation 216 such diat the input and 
output of both the difference and sunoming elements are 
equal. If delta images are to be generated then select<n' A 
selects the ou^ut of the image store. Data moves according 
to the diagram to produce compressed ou^ut images and to 
iqxlate die contents of the image store. Note that decom- 
pressed images entering the image store are identical to 
those that will be reproduced by an eventual decompression 
fiinction. The data flow diagram of FIG. IIB depicts com- 
pression of a key image and three delta images according to 
the process of FIG. IIA. 

llie interfirame compressor of FIG. IIA carries history of 
the video sequence being compressed in the image store. To 
implement video compression of multiple, unrelated, image 
streams therefore requires a separate compressor of the type 
shown in FIG. IIA for each video stream Such multiple 
image streams are generated by multi-camera surveillance 
applications. Correlation is high in the image stream firom a 
single camera, but low between the streams from various 
cameras. 

An alternate approach to multiple stream coni^ircssion is 
to share a single compressor between video streams and to 
maintain a separate image st<H'e for each stream. This 
£^>proach is depicted in FIG. IIC. When compressing a delta 
image for a video stream 1. for example, selector A 210 
selects the previously stored image associated with video 
stream 1 from image store 1 208-1. At completion of the 
compress cycle, selector B 212 passes the newly decom- 
pressed image associated with video stream 1 to image store 
1. 
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The multi-channel compressor can be pipelined, unlike 
the single-channel compressor of FIG. UA. FIG. IID illus- 
trates the pipeline pinciple with the addition of a second 
selector A' 214 at the output of the multiple image stores. 

5 With this arrangement each block in die diagram, difference 
element 200, encoder 202. decoder 204, and summing 
element 206 can be assigned a different image to process 
without waiting for data from the previous image to com- 
plete propagation through the loop. Assume, for instance, 
that each of the aforementioned elements requires one 
execution cyde to process an image. A single execution 
cyde is postulated for illustration where: difference element 
200 processes an input image from video stream 4; encoder 
202 processes a delta image assodated widi video stream 3; 
decoder 204 processes a ddta image assodated with video 
stream 2; and summing element 206 processes a delta image 
assodated widi video stream 1. Seleaor A' 214 in this 
instance couples output from image store 1 20S-1 to the 
summing element; selector B 212 couples ou^ut of same 
summing element as the new value for decoded image 1 into 

20 image store 1. Selects A couples output from image store 4 
(not explicitiy shown) to the difference element 
File manager 

FIG. 5 summarizes data flows for the record function. 
Compressed image data is passed to a file manager 160 

25 which appends date and time codes provided by a time/date 
generator 162. Camera source codes and codes for decom- 
pressing images are included with the data by way of the 
capture and compression stages. The file manager also 
receives non-image data induding digital audio, alarm 

30 triggers, and other data that can be rq>resented in text or 
binary form, appends date and time codes, and organizes 
them for writing to the storage devices. 

One CH- more storage devices store the conapressed image 
data in a format independent of the data type. That is. the 

35 storage device format is not for example, based on block 
sizes assodated widi video fidds or flames. In a first 
embodiment of the invention, data is temporarily stored on 
a fixed-media storage device 78 (e.g. a magnetic disk drive) 
and then is transferred from this fixed-media storage device 

40 to a removable-media storage device 76 (e.g. instrument 
tape drive or removable-media disk drive). This transfer 
occurs at a rate condudve to effident mechanical utilization 
of die removable-media storage device. The fixed-media 
device serves, in effect as an input buffer for die removable- 

4S media device, ff a t^ drive is used, for exan^le, as the 
removable-media device then the tape runs only for short 
bursts to transfer blocks of data at maximum rate, and is idle 
the remainder of the time. This airangemeot is most efficient 
for storing variably-compressed data since there is no need 

50 to link the compressed data rate to a constant tape speed. In 
playback, the above process will be reversed as data is 
transferred from tape to disk and then read from disk at a rate 
to match playback display. 

A second embodiment of die invention uses only a single 

55 mass storage device, employing for exan^le a fixed-media 
storage device such as a magnetic disk drive. From this 
embodiment a completely automatic, unattended recording 
monitor system which needs no routine media changes, 
deamng, or maintenance can be devised Hxis unattended 

60 recording monitor overwrites older data witfi new data 
according to a replacement process. The replacement pro- 
cess causes images recorded across a most recent interval of 
time called an archive interval remain available on the 
storage device for review or copy. Rate control techniques 

65 discussed elsewhere in this specification assure that data will 
not be written at a rate such that archive limits would be 
violated or that new data would be lost 
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Blocks of images recorded on the second embodiment also retrieve data sets from removable-media storage 76 or 

device may also be placed on hold such that they will be fixed-media storage 78 for copy to the rcmovablc-mcdia 

preserved indefinitely and not be overwritten. Block(s) can transfer drive. 

be designated for holding according to facton such as date, RATE COSTROL TECHNIQUES 

time, and camera source. Block holding would be used, for 5 ... .i * ^ 

«a^e, if an event occurs within a Lown time window . Of^ators of suryeiUaDce systems generaUy P^eferjecord- 

^^es needed to be preserved for later review or for ing devices that fiU the storage media in predictable time 

tran^outside the recording device. Block holding, of penods. In a removable-mj^ system-.>ne 

course, reduces available spa^ on the storage device'llie t^pe cassettes, for example-tt is ^^J^^^^J^-^ 

\T.?ovttS^n^:^^^^^ ^^-gcd^y the oper^or on a regular s^^J 

ofseveralpossibledatareplacementpcocesses.Replacement ^^^^^ours) and saved for an ^^^^^^ 

process A diown in HG 12A opiates according to the '^^^ is the interval of time daU must be retained after the 

simple nile that new data always overwrites the oldest data 15 tmie of recording. 

item on the storage device. This jHOcess has an advantage in In a fixed-media system— one that lecc^ds onto a hard- 

that an unbroken data recwd will be maintained beyond the disk drive, for cxan^le-whcre o\6a data ^e continually 

archive interval Tlie disadvantage of this process however oven^mttcn by new datajt is desued that da^ 

is that any fragmentation of data on the storage device is a minimum erf the specified archive interval. Thus, any given 

perpetuated Data fragmentation can occur when storage 20 Poi^t on the media should not be overwritten more fre- 

Es are placed on hold or when the operating system, for quendy than once per archive interval. In this case therefore, 

some reason, does not write data at the next physically the recording fill lai^get interval, or time to fill the storage 

available location on the storage device. «^edla once, is equal to the specified archive interval. 

Replacement process B (FIG. 12B) defines a storage The goal, whethei the storage media is removable or 

location pointer to represent physical locations on the drive. 25 fixed, is to filL or nearly fill, the media in the fill target 

This pointer carries the value of the location last written Interval. IdeaUy. the media would never become filled pnor 

when process B is entered The process first increments die to the end of the fiU target interval such that important data 

pointer to the next physical storage location. If data occu- would be lost Stated another way, tiie goal is to provide a 

pying that location is newer than the specified archive rate of coverage, in terms of recorded image rates and 

interval then the storage pointer is incremented again and the 30 resolution, that is both the maximum attainable for a given 

foUowing location is checked This process continues until storage capacity and consistent, or without large 

a location is found that can be overwritten without violating fluctuations, across the entire receding interval. Loss of 

the archive mandate. Process B therefore fills the storage data near the end of the interval due to fuU storage media is 

device sequentially, except when data items protected by the an example of inconsistent coverage (see example shown in 

archive interval are encountered. Data replacement accord- 33 FIG. 16). 

ing to this process will tend to defragment the storage At least two factors contribute to variability in data flows 

device, provided that storage is mnning at less than ftiU and media filling rates in a compressed video recording 

capacity (defined as not all data items newer than the archive device. One is the inaease in the compressed video data rate 

interval). occurs when images with larger motion content are 

Other replacement processes are possible that combine 40 compressed by the intcrframc image compressor. A second 

the principles of replacement processes A and B. For factor, which is independent of data compression, results 

instance, replacement process A could be adhered to until from accelerating the receding rate in response to event 

stOTage device fragmentation exceeds a certain threshold triggers. This technique is sometimes applied in order to 

Replacement j^ess B could then be invoked until the capture more information during a possibly important event 

storage device becomes sufSdently defragmented Main- 45 Both factors are functions of activity at the recording sites 

taining storage records in a defragmented state can reduce and as such, cannot be compensated for by initial madiine 

mechanical wear on tfic storage drive. A magnetic hard-disk progranmiing. Left uncorrected the variable data flows 

drive, for example, is generally assumed to last longer if would cause irregular and unpredictable fill times for the 

head seeks to different areas of the disk platters are reduced storage media. 

FIG. 7 summarizes signal flows for the playback function. 50 The invention solves the problem of irregular media fill 

A playback controller 174 directs which images or sequence times in a compressed video recording system by making 

of images are to be decompressed and displayed. Motion variable the amount of image data which is recorded and by 

images must be rq)roduced in step with a clock to recreate applying a control system to adjust this amount of image 

motion as it was recorded The playback controller directs a data in order to manage the filling of the storage media. This 

data searcher 170 to fetch images from storage for the 55 solution will be referred to in general as a 'rate control 

requested sequence. These images arc decompressed by system*. 

image decompressor 172. The playback controller loads Mathematically, the compressed video data rale is equal to 

decom^aressed images into video converter 142 or graphics the image rate multiplied by the data per image. Using 
adaptor 140 in step with a playback clock derived from relations 1 and 3. we find: 

system clock 106. The playback controller may cause the 60 

video converter or graphics adaptOT to repeat images until a image rate x (relation 4) 

new image is available from die storage record The play- compresskm data rate - ^^^^^^^ 
back c(Hitroller, image decompressor, and data searcher may 

be in^lemcnted by software executing on the CPU and as In the above context, data rate is expressed in quantities such 

such do not qjpear on the hardware drawing of FIG. 6. 65 as bits per second, and image rate in images per second. The 
Removat>le-media transfer drive 80 is accessed by the conq)ression ratio is a function of image complexity factors 
data searcher as a playback source. The data searcher can such as detail and motion. 



09/01/2004, EAST version: 1.4.1 



5J24,475 

17 18 

The control system operates to modify the image record- size parameters, which are determined by the control block, 

ing rate and/or the image resolution to con^osate for can reduce the rate of data transferred to the storage media, 

changes in comjiressed data rate caused by factors such as The output of the image array opexator is a processed data 

motion within the field of view* variations in image detail stream 324 which is directed to the image compressor 74. 

and recording changes driven by event triggers. In response 5 The image conqyressor outputs a storage data stream 304 

to these factors, the control system seeks to maximize, and which is directed to the storage sub-system 306. 

make uniform, image recording rate and resolution given the Parameters sampled by the control block 350 include the 

limitations imposed by a finite storage space for data and the storage data rate 312 and the storage data volume 314. The 

requirement to support a particular fill target interval. volume of data in the storage media is . of course, the integral 

FIG. 13 shows a fundamental rate control system accord- 10 of the storage data rate taken from the beginning of the fill 

ing to the jHindples described above. Input images from a interval, as will be shown later in relation 6. Therefore a 

video image stream 300 are api^ed to an image processcr control block can be devised that monitors only one of the 

302. The image processor represents all processing fiinc- parameters, either storage data rate 312 or storage data 

tions to be carried out prior to data storage including volume 314 and infers the other by calculation. The control 

digitization, data reduction, and image comf^ession. The is block acts on the image rate 332 and array size 334 inputs 

ou^ut of the image processor is a storage data stream 304 to the image array operatcr to control the rate of the storage 

containing the processed video images. The storage data data stream as before. 

stream is directed to a storage sub-system 306 for storage. The image compressor 74 in FIG. 14 represents an 

The storage sub-system contains the storage media on which additional point in the data path where the rate of the storage 

data is stored. A control system 310 includes capability to 20 data stream can be both measured and altered. The rate of 

monitor storage data rate 312, which is the rate of data in the data in the storage data stream is inversely proportional to 

storage data stream. The control system also monitors compression ratio. Higher compression ratios tend to intro- 

attributes of die storage sub-system including storage data duoe larger compression errors, or losses of fidelity between 

volume 314, which is the volume of data accumulated in the original and eventual decompressed images. Also, 

storage. The image processor 302 receives control inputs fcr 25 greater input image conqilexity tends to force conopression 

varying parameters of in:iage processing. These inputs errors higher unless the compression ratio is lowered, 

include image rate 316. and image resolution 318. Each of CoJI^lression ratio may be influenced by the compression 

these inputs connects to an output from the control system target 340. which is an input to the image compressor. 

310. Several strategies exist fw controlling the compression 

The control system 310 varies parameters of the image 30 target First, image complexity can be measured ahead of die 

processor 302 in order to control the data rate of the storage image compressor by an input concplexity gauge 336. The 

data stream 304. By controlling the storage data stream the resulting image complexity 338 data is ^plied to function f 1 

control system is able to cause die storage sub-system to fill 344 of the control block. The output of function fl in turn 

in a desired manner. controls the conq^ression target 340 parameter. A second 

The remainder of this section first describes control 35 strategy is to measure the compression error 342. Compres- 

systems that operate without feedback. It then introduces a sion target is then adjusted by way of function f2 346 to 

basic and then an enhanced feedback control algorithm for maintain a desired compression error. A third strategy is to 

controlling single-pass filling of storage media. A basic monitor storage data rate 312 and adjust the compression 

control algoriduQ is set up singly by invoking the two target by way of function f3 348 to maintain a desired 

parameters of fill target (equal to storage capacity) and the 40 storage data rate at the expense of compression error, 

fill target interval to control the rate at which data is sent to Functions f2 and f3 represent feedback paths in contrast to 

storage. An enhanced control algorithm makes use of addi- function fl which represents a feedforward path. As such. f2 

tional parameters such as a set of predetermined data rate and f3 can be implemented either iteratively or cumulatively, 

estinoates or profiles. Data rate profiles improve control An example of iterative feedback is multiple pass compres- 

performance by anticipating the typically recurring patterns 45 sion of a single image with varying compression targets 

in the rate significant data encountered at most monitored controlled via 12 until the desired concession error is 

sites. Finally, the basic and enhanced algorithms are adapted attained. An exanq)le of cumulative feedback is single pass 

to the case of continuously-overwritten storage in fixed- compression of images and adjustment of conapression 

media recorders. target via f3 according to the result of one or several prior 

FIGS. 14 and 15 illustrate particular rate control systems so images, 

derived from FIG. 13 in which digital and analog video With the foregoing in mind, it is apparent that there are 

inputs are accepted. FIG. 14 also shows how parameters of several possible methods of inferring storage data rate for a 

compression can be incc^porated into the rate control sys- particular compression implementation. Image complexity, 

tern Actual ai^iications may utilize fewer means than compression target, compression ratio, and con^^ression 

shown in any of FIGS. 13, 14. or 15. 55 error can each provide infcHinatioD for driving a rate control 

Id fig. 14 the image processor 302 of FIG. 13 is broken system. It is also apparent that the st(»-age data rate can be 

into the two separate functions. These functions are an influenced by altering the ooii^>ression ratio by way of the 

image array operator 322 and an image compressor 74. A concession target parameter. Thus, image resolution and 

digital video image stream 320 is first applied to the image image fidelity can be traded to control the rate of the storage 

array operator. The image array operator passes a partial 60 data stream for the purpose of rate control. 

numbCT of the whole images available at the input under FIG. 15 illustrates a rate control systems for analog video 

control of the image rate 332 input The image array operator inputs. Details for monitoring and controlling parameters of 

can also reduce the pixel density of passed images either by compression are not shown in this figure, though they could 

respacing pixels through interpolation or by direct elimina- be applied as were shown in FIG. 14. In FIG. 15 an analog 

tion of pixels. This array reduction operates under control of 65 video image stream 360 is connected to a video digitizer 72. 

the array size 334 input. Reducing pixel density effectively The video digitizer digitizes a partial number of the whole 

lowers the resolution of images. Both image rate and array images in the analog video image stream under control of the 
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image rate 332 input The pixel density of the resulting thereby resulting in the exact filling of the storage nacdia 

image array is detoroined by the sampling rate of the video (capacity=100 units) at the end of the fill interval. Of course* 

digitizer. San^iing rate, and thus array size, is controlled by this is an impractical situation, since complete and reliable 

the array size 334 input to the digitizer. The output of the a priori knowledge of data rates is not possible in practice, 

video digitizer is a processed data stream 324. The remain- 5 Nonetheless, it serves to illustrate the key criteria of main- 

der of the system shown in FIG. IS operates in the same taining level record rates and resolutions and thus consistent 

manner as do the systems of FIGS. 13 and 14. coverage of the monitored site. 

Control Without Feedback An obvious method of employing feedback to fill the 

Most video cameras generate frame rates in excess of that storage media exactly on time is to force the rate of data 

required foe typical surveillance recording. In fact, it is lo entering the storage media to be constant, and of such a rate 

usually necessary to process and store only a small firaction that it always matches the storajge fill target divided by the 

of frames from the camera in order not to overextend the fill target interval However, this solution is unacceptable for 

data storage medium. In a system without feedback control, the reason that it would cause the combined image rate and 

camera frames are chosen for processing at a pre-set con- resolution to fluctuate drastically and in inverse pr<^rtion 

stant rate which is generally a fraction of the input camera 15 to the motion content of the images. Image rate would race, 

rate. In a system with feedback control control can be for example, when no motion was present in the recorded 

introduced at several points in the system as described field of view and drop to a rate many times lower when large 

previously. For example, the fraction of camera frames activity commenced. Motion triggers applied lo video 

chosen can be controlled to be greater or less than the pre-set recorders, in fact, seek exactly the opposite effect — that of 

constant In addition, control can be iiiq>osed on parameters 20 increasing the frame rate when activity is present, 

of the image itself — e.g., array size and compression ratio. Basic Control Loop 

The collective eflfect of influencing these parameters for Ac FIG. 18 introduces an example of a basic control algo- 

puipose of rate control is represented by a quantity now rithm implemented within flic control block. This example 

introduced called the "data rate modifier.'* A value of unity uses the rate modifier concept to simplify rqtftsentation of 

for the data rate modifier means that the c<^ective action of 25 image rate and image resolution control as discussed earlier, 

the control variables is the same as for their pre-set, or The rate modifier modulates, or multiplies, a hypothetical 

nominal, values as programmed by the designer or operator base data rate, b(t). resulting in the storage data rate, s(t). The 

of the recording device. Data rate modifiers greater or lesser base data rate is the rate data would enter storage were flic 

than one mean that the control feedback system has caused control loop to exert neither an accelerating nor a dccclcr- 

the collective action of the control variables to be greater or 30 ating rate influence. 

less than their nominal values. In die later discussion, it is In FIG. 18 the video image stream 300 is applied to a 

brought out that the data rate modifier can vary as a function image processor 370. The image processor in the figure 

of time, and is accordingly given the symbol r(t). accepts a rate modifier 374 as the single control parameter. 

Illustration ofratc control examples begins with a system The rate modifier lumps together the effects of varying 

in which the rate modifier r(t) has been set to a constant 35 image rale, image resolution, and conqiression parameters as 

value of unity. This is the case where no feedback control is described above. The output of the image processor is a 

i^>pUed and all data rate influence are set to constant, storage data stream 304 which is directed to the stOTage 

nominal values. FIG. 16 shows a time history for this sul>- system 306 as before. The storage sul>-system outputs a 

example in which a storage data stream of variable data rate measure of the storage data volume 378 which is applied to 

is filling the storage media. 40 the control system 310. A data rate gauge 372 monitors the 

The storage data rate — the rate at which con^ressed data storage data stream and outputs a measure of the rate of this 

is being sent to storage— is shown across the fill target stream as storage data rate 376. Storage date rate is applied 

interval in FIG. 16. Even though factors such as image rate to the control block. Storage data volume is represented by 

are constant as determined by the rate modifier, ttie stwage the symbol v(t), and storage data rate is represented by the 

dato rate nevertheless fluctuates. The fluctuation is a result of 45 symbol s(t). 

image motion and detail changing with time — i.e., when The main control element in the loop is an integrator 388 

there is significant motion in the fields of view of the that serves to smooth the response to short-term fluctuations 

surveillance cameras, the storage data rate necessarily rises. in compressed data rate while providing sufSdent long-term 

In this example, this rate fluctuates randomly over approxi- correction to fill the storage media acceptably close to the fill 

raately a 4:1 range. Storage data volume — the accumulated 50 time target, T. The output of the integralOT is the rate 

daU in the storage media— is also shown in the figure. modifier 374. The rate modifier is represented by the symbol 

Storage data volume can be gr^hically calculated by find- r(t). Tlie integrator is driven by two quantities. One is the 

ing the area under the storage data rate line from t=€ to the difference between the storage data rate 376 and a value 

abscissa. The fill target is set equal to the storage capacity of corresponding to an "idealized" data rate of C/T 384. This 

100 units in this example. The rate modifier. r(t), which, as 55 difference is generated by difference clement 380 and is 

mentioned above, is set to a value of 1,0, is similarly shown. scaled by the factor k2T/C. The second quantity is the 

Unfortunately, this value of rate modifier results, in this dijBfcrence between the storage data volume 378 and a value 

example, in the storage media becoming filled after only corresponding to an 'idealized** data accumulation of t*C/r 

85% of the fill interval has elapsed — i.e., with 15% of the 388. This difference is generated by difference element 382 

time interval still remaining, and no place for the remaining 60 and is the scaled by the factor kl/C Constant C is the fill 

data to go. target for the storage media. Constants kl and k2 set loop 

FIG. 17 illustrates the same non-feedback situation as performance. 

FIG. 16. except that the data rate modifier has been lowered At the start of a cycle of operation. r(t) is set equal to 

to a value of 0.803. The storage data stream is accordingly unity. As the cycle proceeds, the integrator will adjust the 

lower by the same amount over that of FIG. 16. The value 65 value of r(t) to more nearly optimize the storage utilization, 
of 0.803 has been chosen based on a priori knowledge of The algcnithm of FIG. 18 is equivalentiy expressed by 

what the data rate history will be over the next fill interval, relations 5 and 6 below: 
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because of motion. In response to this however, the rate 

/</) = 1 + (pclatioa 5) modifier drops by only 12%. This means the loop faces 

down the rate that images arc recorded by only 12%. The 

r f rest of the base rate increase is accommodated in storage as 

I ((U/c^f*c^-v)) + (*2*r/cnoT-j))V* 5 extra accumulated data. Near the end of the fill interval, the 

0 base data rate again (unexpectedly) doubles from time t=80 

*<r) = rit)*H^ ^ ^ interval Base data rate increases near the 

end of the fill interval would be difficult to acconunodatc. 

and except that since fill target, C. has been set to 90% of 

capacity, the unexpected data can be stored. The final result 

r r ^"^^ ^ is that data fills 96.8% of the storage media capacity— in 

= J tdt excess of the fill target of 90 units but well within the actual 

° capacity of 100 units, 

where variables: The example of FIG. 20 uses the same loop settings as the 

t is time example of FIG. 19, but assumes the base data rate shown 

bit) is base data rate, or data rate prior to loop correction ^p^. 16 and 17, which is more representative of an 

sudi that sftWrt'i*Wt^ ictaal apphcanon. Once again it can be seen that, because of 

. J r * j4 u -u • *u • u. the integration in the control loop, the short-term fluctua- 

r(t) IS the data rate modifier, which is the vanable gener- ^^^^ ^ ^^^^ ^3^^ ^ ^^^^ 

ated by the control loop modifier. The longer-term increase in base data rate 

s(t) is the storage data rate— <hc rate at which conqjressed 20 apparent in FIG. 16. results in the rate modifier value being 

data is sent to storage adjusted downwards aaoss the interval in HO, 20 to prevent 

v(t) storage data volume — the volume of data accumu- storage media overflow. Final data storage exceeds fill target 

lated on the storage media by only 2% in this example; there is still an unused margin 

and constants: of 8% of the storage media capacity. 

C is the fill target — the target aixK)unt of data to be stored 25 Rate Profiles 

on the storage media during time interval T The performance of the basic control algorithm of FIG. 18 

T is the recording fill target interval to store the data, C, can be improved if some realistic estimate of data rates can 

on the storage media. T is usually chosen to equal the be provided in advance of the recording cycle. It is particu- 

ardiive intervaL larly useful if the estimates can be related to clock or 

The basic control loop described by FIG. 18 and relation 30 calendar time— 4. e., time of day or day of week. In this 

5 is a second-order loop. The second integration is embodied section, we discuss how information of this type, assembled 

in the operation of filling the storage media, as shown by into what we call a "^profile," can be used in an Enhanced 

relation 6. A second-order loop is employed so that the error Control Loop to improve the perfomiance of the system 

between the storage data volume v(t) and its target value of A survey of monitored site applications reveal that motion 

C*t/T will approach zero for a steady-slate base data rate 35 activity, and therefore compressed data rates, typically fol- 

b(t). Rate feedback, set by coefficient k2, stabilizes the low daily and weekly patterns. For example, a video rccord- 

double-integrator loop. ing system monitoring an industrial plant cr office staffed 

The delay inherent in the integrator response permits primarily by a weekday workforce might generate a base 

tenqxjrary increases in compressed data rat e^— caused by data rate, b(t), of the type shown in the seven day graph of 

unexpected image motion, for example — to be accumulated 40 FIG. 21. Here, the data rate exhibits a pattern of te-oad peaks 

in the storage media without immediately lowering image during the weekday hours of 7AM to 6PM. This results from 

rate to compensate for the extra data. As a consequence, the the majority of the workforce being |»esent at die facility 

loop cannot prevent filling past the fill target if these during these times. In finer detail, each of ^e five weekday 

unexpected increases in data rate occur near the end of the broad peaks is punctuated by sharper peaks coinciding with 

fill interval, T. It is therefore prudent to choose the c^dty 45 activities such as lunch, work breaks, and the arrival and 

of the stOTage media to be greater than the fill target C. The departure of personnel. Smaller, irregular peaks may occur 

margin of capacity over fill target is chosen as a function of on evenings and weekends as a handful of employees and 

the loop response speed and the anticipated wo-st-case step security staff dieck in and out of the facility. A certain 

inaease in base data rate. mini mum base data rate manifests itself at the output of the 

The example of FIG. 19 examines the response of the 50 image compression system even when no motion content is 

basic control loop to step inputs. Here, the value of the fill present in the scene. This minimum rate results from camera 

target C, has been set to 90% of the actual storage cq)acity noise, image background fluctuations, and protocol data 

of 100. This provides a filling error margin of 10%. In this identifying advancement of the image sequence, 

example, the base data rate. b(t), steps between the two FIG. 22 introduces the enhanced control loop concept by 

levels of 0.8 and 1.6. Loop constants kl and k2 are set to 55 adding two components to the fundamental rate control 

0.15 and 0.05, respectively. The base data rate, which is 0.8 system described in FIG. 13. These two components arc a 

until time step t=45, is sli^tly below the target value for the rate iH-ofile accumulator 390, and a clock/calendar 3!>8. A 

storage data rate, s(t). of 0.9 (C=90 divided by T=100). store of pre-programmed rate profiles 396 can optionally be 

Therefore in accordance with the algorithm, the loop slowly included. The rate profile accumulator receives the informa- 

inaeases the rate modifier, r(t), above the initialization value 60 Uon of storage data rate 312 and storage data volume 314. 

of 1.0. By raising r(t) the loop is acting to increase the rate These parameters are monitored by the control system 310. 

of data entering storage by. for example, processing a greater The rate profile accumulator also receives infonnation of 

number of camera images than were originally required by image rate 316 and image resolution 318. These parameters 

initial conditions. are outputs of the control system. Finally, the rate profile 

In the time interval t=45 to t^8 the base data rate briefly 65 accumulator receives time/date data from the clock/calendar 

doubles to 1.6. This represents the case, for example, where 398, and accepts profile data from the store of prc- 

the compressed data content of each image has doubled programmed rate profiles 396. 
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The purpose of the rate profile accumulator is to observe where applicable. For example, if a storage media is to be 
the data rates over a period of time, and by averaging this filled over a 24 hour interval from 9AM on TXiesday to 9AM 
activity over a number of time periods^-days, weeks or on Wednesday then a rate profile is assembled for this 
months — develop a rate profile. FIG. 21 is actually such a Interval by recalling 52 prior examples of the interval 
rate iH-oftle. showing data collected over a number of weeks, 5 Tuesday 9AM to Wednesday 9AM from the year-long 
and averaged to develop the curve shown. Pre-programmed record of base data rates. The 52 segments are averaged 
rate profiles include all profile data that is not accumulated together to produce a rate profile whidi projects the likely 
through the raonitcring of instrument record cycles. Pre- ^^di 15-minute segment of ttie upcoming 
programmed profiles are externally p-c^ammed data, eitocr Tuesday 9AM to Wednesday 9AM interval. Statistical tech- 
programmed during design of the instrument ot locaUy lO ^^^^^ ^ ^ ^ ^^^^ confidence in the profile, 
progranuned for the particular site of operation. Anomalous segments can thus be detected and left out of the 

The ou^ut of the rate profile accumulator is a pair of averaging process 

signals. These signals ref^esent estimates of the storage data \^ ' ^, u ^ * * u /•x 

rates and storage data volumes to be encountered in a new ^G, 23 shows aprofile base data rate, bp(t), generated for 

recording cycle. The first signal is the i^ofile storage data 15 a 24-hour interval according to tfie methods described 

rate 394, which is represented by the symbol sp(t). The above. The Une for bp(t) shows high base data rates during 

p-ofile storage data rate is used by die control block as a the hours of 9AM to approximately 6PM caused by the bulk 

target for the storage data rate during the new recording of the daytime workforce being on location during those 

cycle. A second signal is the profile storage data volume 392, times. The low base data rates from 6PM until approxi- 

whidi is generated by integrating the profile storage data 20 mately 7AM the following nKaning occur while the plant or 

rate from the start of the recording interval. Profile storage facility is fairly empty. High base dau rates resume at 7AM 

data volume is represented by the symbol vp(t), and is used when the next workday commences. Data accumulation, 

by the control block as a target f<ff staage data volume ^vhich is generated by integrating bp(t) from the start of the 

during the recording cycle. interval, is also shown. In this particular example the data 

The profile storage data rate. sp(t). and profile storage data 25 accumulation curve terminates at a value of 128 at die end 

volume, vp(t), signals serve as tenq)lates of storage media 24 hour interval 

filling to aUow the enhanced ^"^^^^^'^.^^^ ftofilcs sp(t) and yp(t) are generated &om the curves of 

stable image rat« and nssoluUons against tte vmaWedata ^ ' « accumulation matches 

flows encountered m a real reoardmg cycle. This IS based on 'Tr ^ ^ - 1*1 i- 

L assumption that data patterns in a new recording cycle 30 the fill target, C at toe end of the intervaLAs in an earher 

wiU reserSwe, to a degree, the data patterns that occurred in c^lc* ^ target of 90% of the storage media capacity of 

carUer record cycles. The enhanced control loop thus exer- 100 units is chosen. The curves of HG. 23 must Uiercfore be 

rises minimum control influence by way of the rate modifier scaled by 90/128. or 0.702 to cause 90 units of data to be 

output variable when data patterns in the record cycle match accumulated by the end of the interval The result is a set of 

th^of the profile cycle. The daU profiles en^loyed by the 35 curves for profile storage data rate, sp(t), and {w-ofilc storage 

enhanced loop contrast to the constant value data rate and data volume, vp(t), shown in FIG. 24. A third line has been 

linear-time storage accumulation targets {CJT and fCTT) added to show the scaling factor of 0.702 which is constant 

employed by the basic control loop. As before, the objective across the interval. This scaUng facto: can also be mta- 

is to maximize the nKxint of data transferred to the stwage F^ed as the i^ofilc rate modifier, rp. since it would repre- 

media in a predetermined interval without large rate modi- 40 sent the optimal rate modifier value for a rccordmg interval 

fier fluctuations and without exceeding the capacity of the wi& the given prc^e base data rates. As will be seen later 

storage media such that subsequent data would be compro- this value, rp, can be used as a starting, or imtializauon 

mised or lost value, for the rate modifier in enhanced loop operation. 

The rate profile accumulatcM' shown in FIG. 22 collects Sununary 1: the steps taken to create rale profile curves for 

and maintains a record of base data rates observed in 45 a new recording cycle are 

operation of the recording device over an extended period erf Determine which time segments in the existing record 

time. Note that base daU rate is defined, as before, as the rate ^^^^ ^ anticipated to conelate with the 

of compressed video data produced by the system when it recording interval- 

operatesaccordingtosomedefaultorpreset.imagerateand ^ 

resolution parameters to the exduston of any mfluence by so ^^^f^^ ^^^^ ^ 

the rate control loop. This base data rate must tiiercforc be wI^wKi^tulvf^ ^^ftw/n hut^ ^hrnnoh 

inferred indircctiTby the rate profile accumulator since, bp(t)=:[bl(t>+b2(tH- . -rtm^l/n where bl t) through 

imciicu luuutvt*/ u A bn(i) represent n base data rate segnicnls selected 

whUethecontronoop«oporating.thebaseMe^^^ fro^STrecordof base data rates in step 1. 

exist as a du-cctty measurable quantity in the system The *^ 

rate profile accumulator therefore measures actual data rate 55 3) Scale this profile base data rate. bp(t), by a constant rp 

into storage s(t) and takes account of the effects imposed by to create a profile storage data rate curve, sp(t), and a 

the rate modifier r(t) on image rate and resolution to calcu- profile staage data volume curve vp(t) such that the 

late tiie profiles according to bit)=s(tyr(t). The resulting storage volume at the end of the target interval, T, 

assessment of base data rate can be stored as a scries of equals the fiU target C. Specifically rp, sp(t,) and vp(t) 

bytes. For example, a byte can represent a 15-minute aver- 60 are defined so that; 
age of operation. The record for a complete year could then 
be retained in only 35 kbytes of storage. 

A rate profile is generated from segmcnU of the hasc data 
rate record that best conclate to the recording interval of 

interest. CcHTelation is most likely to be found across daily 65 spit) = rp^t) (reiaiwo 8) 

and weekly patterns, although correlation to monthly, 
seasonal, or other time intervals could be examined and used 



(relatioD 7) 
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-continued presumed similar to the profile cycle the value rp determined 

(reUtioQ 9) earlier for the profile cycle is assumed as the best initial 
setting fcff the rate naodificr r<t), Ihc rate modifier r(t) 

0 accordingly begins the interval with a value of 0.702 in FIG. 

and, therefore; ^j^^, i«cording cycle proceeds, the rale modifier r(t) 

is driven by two mor signals. The first error signal is 
proportional to &e difference between actual and profile 

Enhanced Control Loop Utilizes Rate Profiles storage data volume (vp-v) This difference can be inferred 

The enhanced control algorithm is based on a modified- from the gap between the curves for vp(t) and v(t) drawn in 

don of the basic control algorithm {deviously described by FIG. 27. The second error signal is proportional to the 

FIG. 18 and relation 5. This new algorithm which is difference between actual and profile storage data rates, also 

described by FIG. 25 and relation 10 incorporates the rate apparent in FIG. 27. These error signals are scaled as 

profile data rq)resented by variables ip, $p(t). and vp(t). described in relation 10 and applied to the integrator sup- 

Spcdfically, relation 10 is derived firom relation 5 by sub- plying the rate modifier r(t). The benefits of tiic enhanced 

stituting vp(t) for t*C/r in the kl term, sp(t) for C/T in the loop are apparent firom examination of rate modifier r(t) 

k2 term, and rp for the constant of Integration of one (the values cf FIG. 27. The rate modifier varies only over the 

vahie of r(t) at t=0). range of 0.576 to 0.783 (a range of -18% to +12% compared 

to the initial value of 0.702) while accommodating a base 

- f (relation 10) ^ <jata rate which varies across a ten-to-one range. 

/</) = ip + J ((kyc*(vp - v)) + (k2*T/c*(sp - s)))dt Control for Systems with Overwriting Storage 

0 Continuously overwriting systems are allowed to over- 

whcre variables* write stored data which has been on the storage media fw a 

s(t) is the storage dau rate ^ ^^^^^^^^ 1 1 and 12 

. .t. r. J . * below emxress the quantity of data on the storage device. For 

sp(t) IS the profile storage data rate 25 ovcnvriting storage the archive interval equals 

v(t) storage data volume— the volume of data accurau- ^ interval. This is so because overwriting data 

lated on the storage media per archive interval is equivalent to filling the media 

vp(t) is the profile storage data volume once per archive interval. Both fill target and archive interval 

r(t) is the data rate modifier, which is the control loop ^ will be represented by Ac variable T. 

ou^ut variable 

rp is the rate modifier calculated for the rate profile f t (reiatioD U) 



b(t) is base data rate, or data rate prior to loop CGcrection 
such that 

i<i)=f<f)*iKO 



v</)=| sdt: for t<T 
0 



= 1 jdSf- I j(t~'l)dt:fort>T 
Jo J T 



33 , , ^ . (relation 12) 

K0 = 



and constants Relation 11 represents the initial interval, 0<=t<=T. during 



C is the fill target 40 



which the storage device is filling for the first time. The 
operation of the system during this interval is identical to the 



T is the recording fiU target interval single-pass fill cases described earlier. Relation 12 adds tiie 

The enhanced control algoithm shown in FIG. 25 is ^(t.T) to ex|wess removal of data which is older than 

identical to the basic control algorithm shown in FIG. 18 archive interval T. For operation after time T new data is 

with the exception of inputs 384 and 386 appearing in FIG. tfie storage device whUe older data is simulla- 

18. These inputs have been replaced by ^t) 364, the joofile 45 ^eously removed (mere accurately, space for the older data 

stOTage data rate, and vp(t) 366, the profile storage data de-allocated as described in relation 12. The quantity of 

volume. stored data ideally reaches and maintains a steady-state 

The purpose of the enhanced loop is to use rate profiles y^iuc close to hiU capacity, witfj some margin maintained as 

representing information fi-om prior recording cydes to described earlier as a cushion against rate fluctuations, 

anticipate data flow patterns in a new recording cycle sudi 50 ^^^^ ^^^ntrol loops (tiiose not using rate profiles) for the 

that smaller adjustments can be made to the loop response overwriting storage case are considered next One control 

parameter r(t). In other words, the closer the prestored methodology fcM* overwriting storage seeks to maintain a 

profile matches the new cycle the less action is undertaken constant quantity of data in the storage media. This type of 

by the loop as the new cycle proceeds. j^op is described by relation 13 in which the kl term 

FIGS. 26 and 27 show an example of enhanced loop 55 compares accumulated storage, v(t). against the fill target 

operation. The base data rate, b(t). that will be encountered constant C* and die k2 term compares incoming data rate s(t) 

in a new recording cycle appears in FIG. 26. Also shown is ^jjj outgoing rate s(t-T). 
the profile base data rate. bp(t), which will be engaged as a 

template for this new recording cycle. This profile base data r($) = i+ (reUtioD i3) 

rate is identical to the one appearing in FIG. 23 and is similar 60 

to, but does not exactly match, the base data rate b(t) of the ^ , 

new recording cycle. I {kuc*(c~v)-^k2*T/c*is(t-T)-j{t)))dt:fott>T 

FIG. 27 shows how the enhanced control loop adjusts the ^ 
rate modifier r(t) during the new recording cyde to fill the 

stcH-age media to the target of 90 units. Curves for profile 65 Such a loop operates on the assumption that if data inflows, 

stcrage data rate, sp(t). and profile stcrage data volunoe, s(t). approximately match data ouCQows. s(t-T), then data 

vp(t). are repeated from FIG. 24. Since the new cycle is accumulated in storage, v(t). will remain at or near the fill 
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target C. A problem with this approach arises however, applied to generalize the basic loop of relation 14 into the 

because patterns of data accumulation in earlier intervals enhanced loop of relation 15. Relation 9 is also applied to 

(e.g. t=0 to t=T) tend to affect the data accumulation patterns present vp(t) as the integral of sp(t). 
in the intervals that follow (e.g. t=T to t=2T. etc.). In effect, 

space available to write new data, which arrives at rate s(t), 5 kO = t + (relation 15) 
becomes coupled to the rate, s(t-T), that earlier data is 

released. If data was accumulated at a non-uniform rate ina f'// \ \ 

previous archive interval then this constant storage criteria J I I kuc* J (jp - l + (Ic2»xc*<fp - *)) 1 dt; 1 

tends to cause some of the non-uniformity to propagate ^ \ \ ° / ' 
ahead to the current archive interval. Stabilizing the loop can 

be difficult as well, due to the 'history' effects carried A composite basic and enhanced loop for overwriting star- 
forward by the s(t-T) term. age which operates according to relation 15 is illustrated in 
Better results are obtained simply by extending the single- FIGS. 30 and 31. This 28-day rccOTd interval example is 
pass loop principles outlined in relation 5 as if storage were designed to show how an instrument, first initialized into a 
infinite and as if data were never removed during the entire new site of operation without any prior rate jMrofile data, will 
interval (t>0). We define this new basic control loop accord- pcrfona Rate control during the first seven days will be via 
ing to relation 14 by substituting the single-pass expression the basic control loop. ITus is done by assigning the constant 
of v(t) appearing in relation 6 into relation 5. This is done to value C/T in place of the. as yet unavailable, rate profile 
eliminate die data removal term s(t-T) which is otherwise variable sp(t) in relation 15. Beyond the first seven days of 
present for the overwriting case of v(t) according to relation operation, base rate observations taken during the first seven 
12. days will be pressed into service as a rate profile. The base 

data rate of HG. 28 is used for this example. FIG. 30 shows 

/ y / \ \ (relatioa 14) the ram profile derived from the first seven days of FIG. 28. 

/<!)= I + I I I kvc* { t*aT- f sdt]]+ This seven day rate profile is labelled as sp*(t). For sira- 

Jo\\ \ •'0// 25 Plicity of the example we will use this same seven day 

)pr<^e for each of the three week-long intervals— days 7 to 

^, forr>0 14. days 14 to 21, and days 21 to -28-foUowing the first 
week of the record interval. Assignment of values for sp(t) 
are smrnnarized below: 

The control loop of relation 14 takes into account only the 30 sp(t)=<yr: 

rate of data entering the storage device s(t) and not the rate 

s(t-T) at wbid) older data is removed. Long-term stability of 0<t<7 

this loop is good when kl and k2 arc chosen such that the 

loop settles within the time interval T (choosing setting j^j)_^»(,_7). 

constants kl as 0.15 and k2 as 0.05 provide good settling). 35 

Loop stability can be understood intuitively by noting that if 7<t<14 

the loop settles within die interval T then inclusion of terras 

describing data removal s(t-T). as were done in relation 13, gp(t)=^(t'\4)\ 

will only reintroduce transients that have previously settled 

out The loop of relation 14 operates on the premise that, if 40 for i4<t<21 

the correct amount of data is introduced into stwage during 

new archive intervals then the effects o£ off-target data jp(*)=ip*(t-2i); 

quantities from earlier intervals will disappear as this earlier 

data is removed. for 21<t<28 

FIG. 28 shows a 2&-day base data rate, b(t), example that 45 

will be used to test the following exanqjles of overwriting FIG. 31 show the overall results of the loop operation. The 

control loops. Five peaks per each seven day interval delin- first seven days of FIG. 31 are identical to FIG. 29 since 

eate weekday activity similar to the patterns of FIG. 21. FIG. basic loop operation is in effect. After the first seven days. 

29 illustrates the effects of implementing the basic control fluctuations in the rate modifier diminish as the enhanced 

lo<^ of relation 14 using the base rate of FIG. 28 as input 50 loop utilizing the rate profile takes over. Storage data 

V(t) reaches a maximum deviation from the fill target value, volume, v(t), calculated according to relations 11 and 12 is 

C, of 90 as it crosses 98.5 between days 13 and 14. After also included in the figure. Volume begins to level off past 

that. v(t) settles fairly smoothly toward the target. Rate the first seven days as exfHred data begins to be renaoved at 

modifier fit) fluctuates over a range of 2.75 to 1 as the loop rate s(t-T) according to relation 12. The rate modifier varies 

responds to the low dau rates in intervals between the end 55 only over the range of 1.59 to 1.70 for the last seven days (a 

of fifth weekdays and the start of the first weekdays. Beyond range of -5% to +2% compared to the average value of 1.67 ) 

day 21 the rate modifier has settled to an average value of while the loop acconunodates a base input rate which varies 

1.68 in response to the fact that the averaged base data rate across a ten-to-one range. 

in FIG. 28 is 0.6 of the value required to fill storage to target It should be pointed out that die preceding examples of 

in a fill interval (1/0.6»1.68). overwriting stOTage are functioning with archive intervals, T, 

To make variations in the rate modifier smaller during the of length equal to the periodicity of the profile data. As stated 

recording interval we must employ rate profiles and the earlier, profiles are most likely to correlate, or have periodic 

enhanced control loop. To derive the enhanced loop relation behavior, across daily or weekly intervals. In the given 

for overwriting storage, we recall that basic loops are a examples involving one-week archive intervals and one- 
subset of the more general enhanced loop. The enhanced 65 week data profile periods, the rate that data is stored 
loq) of relation 10 was earlier derived from the basic loop throughout the current week approximately matches the rate 
of relation 5 by substitution. Similar substitutions can be that data, at the same day and time one week earlio", is 
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removed. Thus the quantity of data in storage remains From relations 1. 2, and 3 it is dear Hiat image resolution 

roughly constant This would be true even were the archive is one of the factors that can be invoked in achieving the best 

interval a multiple number of weeks — two weeks or four utilization of available data storage space. For exan^ie, if 

weeks for exaiiple— relative to the one-week proflle peri- images containing relatively little motion content have been 

Q(lj^(y s compressed and recorded early in a given interval, then data 

The overwriting storage rate control techniques described s^rage space greater than the current target value may 

earlier can be expanded to include the case where the archive "-e"?^- Accwdingly. a choice can then be made to retain 

interval is not an integer multiple of the rate profile data fj^^*""^ ^^^^ resoluUOD. either on a nonse- 

period. For example, if a four-day archive interval were used lechve or on a triggered basis. 

rilhthebaserat7showninFrG.28thendatainstorage.v(t). 10 ^ Sdected subsets, or ^wndws^^ of the input iiM^^ 

would tend to peak by the fifth weekday (e.g. day 5) and ^ compressed and saved at high resolution. To demonstrate, 

wuuiu icuu lu peas, uy ulc ului ' » ' WO assumc an input image can be acquired either at a 

drop to '^^^'^ by the first weekday after the weekend ^ ^ dements (pels) or 640>c480 

managing this storage vadaWhty in P ^ 

an overwriting conttol system « to scde the rate modifios J compression are proportional to the origi- 

'^'J^^^^'TS'.*^."?'^^ ^c '^''^ • Sage array size. ej. the comptissSl 640x480 pel imiie 

methods of detennimng profiles In Summary 1 are now ^ ^ 8 

generahzed to include diese cases by replacing relation 7 fLc.-^ itn-yjAn ~i 

with relation 16 below. We define Tp as the rate profile '^'^'ft ^^0x240 pel m«ge. 

period such that: , nG. 32C shows an example m which toe co^^^ 

*^ is acquired at the lower density ofFIG.32A. while a selected 

ipiiy^spiH-NTp); portion — the windowed portion containing the stick 

. figure — is acquired at the higher resolution of FIG. 32B. The 

whaeNis any mtegCT , . • i i resultant can be caUed a spUt-resohition image. In this 

The oroflle rate modifier, rr, is set as the minimum value ... .j • ,j ■ 

iiK iiuiu<^ iai«; luuuu^. ip. w a« « uit^ luuiuu ai w examplc. ttic sUck figure wmdow occupies V4 of the unage 

Of toe expression in brackets m relation 16 under tnal .reHnd is processedas a sub-image ^ 

evaluation of aU values of tin the range of zero to Tp. 25 ^ ^ 

(reUtkm 16) ^ sliDplc technique for compressing and storing a split- 
resolution image is to cany it as two separate images at two 
different resolutions. In this case, the two separate images 

30 would be those of FIGS. 32A and 32D, where FIG. 32D is 

In concluding the discussion of rate control techniques it windowed portion of FIG. 32C redrawn to fuU size. Note 

is pointed out that a number of non-linear techniques can be ^ pjQ 32A carries a low-resolution redundancy of the 

appUed both to inqirove the performance of the rate control information conveyed in FIG, 32D. This is included so as to 

systems and to guard against overflow problems that can maintain the images in a simple rectangular format. This 

arise widi extreme data rams when operating either single- technique requires twice the data of a 320x240 pel image to 

pass OT overwriting control systems. Some of these tech- represent the spUt-resolution image, but is gcneraUy appU- 

niques are outlined briefly below: to ^^^^ encoding algorithm. 

1) Upper limit to rate modifier, r(t), prevents image rate Specific techniques to store portions of a single image at 
acceleration past the point of usefulness when little or different resolutions may lend themselves to particular 
no motion is present ^ image encoding algorithms. DCT encoders such as JPEG 

2) Lower limit to data rate profile. sp(t), such that storage operate on 8xg image Mocks that are independent of adja- 
is allocated at some minimum rate even during inter- cent bloclcs. Different pixel densities can be rq^resented by 
vals when little or no motion is predicted different blocks to build a multi-resolution image. The 

3) Assign non-linear or progressive rate feedback terms structure of wavelet image encoding also makes feasible the 
that become active as stored data levels approach 45 extraction of sub-images from a larger image array, 
storage media capacity. Serves as secondary guard Asanexampleof this second technique, we return to FIG. 
against storage media overfill. 32C. We still save the window as a high-resolution image, as 

DdpriT TTTTrAKT TOi^uMTrkT ICO bcforc. HowcvCT, thc low-fesolution image now omits any 

RESOLUTION TECHNIQUES* ^ windowed area. Multi-resolution images such as 

Unlike conventional videotape recorders, a wide choice of 50 this example can be represented with less compressed data 

image resolutions can be in^lemented on compressed image since the redundancy of carrying the windowed area at low 

data recorders. Since compressed data rates are not linked to resolution is eliminated. Using this technique, the image of 

the turning rate of the recording medium, the resolution and FIG. 32C could be represented with 1.75 times the data of 

resulting data content ofimages can be varied indq)endently a 320x240 pel image with multi-resolution compression 

image by image. Image resolution can be chosen as a 55 techniques. 

function of camera views, time of day, in response to internal Que useful example of split-resolution image recording is 

or externally received triggers, or by any other arbitrary found in siurvciUancc recording of a doorway or entryway. A 

criteria. high-resolution window can be fixed at the location in the 

One method of managing image resolution is to set cntryway where the head of the person entering normally 

resolution according to predetermined criteria to cause 60 comes into view. Thus a high-resolution view of the person 

images from particular camera views, such as entryways, to for identification can be captured along with an overall view 

be recorded at higher resolution than otha camera views, of the entry area. Additionally, an image motion detector can 

Another method is to alter resolution in response to trigger be enabled for the windowed area such that an image is 

criteria. A motion detection trigger, either from an external captured at the time the person is in the windowed field of 

device, or from detection of motion within the image itself, 65 view. If the high-resolution window occupies but a small 

can cause a sequence of images to be collected at higher fraction of the image then the split-resolution Image requires 

resolution. little more storage space than the standard resolution Image. 
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Moving objects can readily be distinguished in successive 
images by evaluating the difference between the successive 
images. A B-A differeoce image is formed by subtracting 
pixel values from corresponding locations in two source 
images, A and B. Zero-valued pixels rqacsent areas in the 
difference image where the two source images are identical; 
non-zero pixel values represent areas where the two source 
images differ. 

It should be remembered that, where an object has moved 
between two successive image views, a B-A difference 
image will contain both positive and negative image com- 
ponents of the moved object That is, both the positive of the 
object as it q>peared io image B and the negative of the 
object as it appeared in image A will be apparent in the B-A 
difference image (to be exact, any appearance of the 'object* 
is actually comprised of the pixel difference between the 
object and the portion of the background image covered ot 
obstructed by the object). 

If the object moves only a few pixels between images, as 
normally occurs when images of human movement are 
collected at 30 frames per second, the two difference objects 
will nearly superimpose as in the illustration of FIG. 33A 
and little positional accuracy will be lost interpreting both as 
a single entity. At common surveillance rates of a few 
images or less per second however, object movement may be 
great between views. In this case the object will clearly 
appear twice and separately in the difference image as in the 
illustration of FIG. 33B. Thus, short of employing elaborate 
image recognition algorithms, examination of a difference 
image similar to FIG. 33B will suggest two indistinguish- 
able objects of motion. 

One method to correctly Isolate a moving object in 
difference images is to track all objects sequentially from the 
time they first enter the field of view and, in an iterative 
manner, track their appearance in difference images as pairs. 
This is done by noting the new pair member which is 
assumed positive and discarding the previously appearing 
pair member which is assumed negative. This method can 
become confused however when multiple objects are present 
and overlapping, and when non-object interference such as 
shadows and lighting changes causes continuity of object 
traddng to be lost 

A prefeired metfiod to distinguish moving objects is 
outlined in FIG. 34. A moving object captured in sequential 
images A, B, and C is represented by a circle in a different 
location in each image. The crosshatcbed areas in images A. 
B. and C represent fixed background detail in the field of 
view. This background detail cancels out when difference 
images C-A and C-B are generated. To identify the location 
of the moving object in image C a position image is formed 
from difference images C-A and C-B as follows. Corre- 
sponding pixels in the two difference images are first com- 
pared. If the pixels are non-zero and match widiin a preset 
threshold then the absolute-valued average of the two pixel 
values is transfored to the new position image. For all other 
cases a value of zero is transferred to the position image. 
In symbolic representation 
if 

C-Aix,y)itO, and C-B(jcy)*CK wid ABSiC-A(x,y)-C'Bix,y)\<^ 
tbresbold 

then 

Pix,y)^ABS[C'Aix,y}*C-Bix,y)] 

else 

P(^y)=o 
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where C-A(x,y), C-B(x,y) represent pixel values oi differ- 
ence arrays and P(x,y) rqiresents a value of the posiUon 
image. 

Note that the value of zero is the same value carried for 

5 non-object backgrounds in the difference images. Thus, 
creation of the difference images first cancels out back- 
ground detail redundant between the original images. Gen- 
eration the position image then cancels out non-matching 
objects brtween difference images C-A and C-B, causing 

10 moving object inq>ressions -A and -B to disappear and 
allowing moving object impression +C to remain. The 
location of +C in the position image corresponds to the 
location of moving object C in original image C. 
A bulk recognition algorithm is next applied to die 

15 position image created in the preceding steps. Bulk recog- 
nition simply isolates and determines the location oi groups 
of non-zero pixels. Criteria must be established as to 
whether single or multiple objects are to be detected and to 
the minimum and maximum window sizes. In the case 

20 where a single minimum-sized window can enclose a suf- 
ficient fraction of non-zero pixels within the position image, 
the recognition is deemed complete and it remains only to 
identify the center of the window so determined. If the 
distribution of non-zero pixels exceeds the minimum win- 
as dow size, then boundaries of a window must be established 
that contain the aforementioned sufficient fraction of non- 
zero pixels. SuflBcient fraction is a threshold value, e.g. 90% 
or 95%. used to discard reflections, sharp shadows, or other 
anomalous effects. 

30 To determine multiple windows for multiple objects the 
algorithm attempts to isolate groups of non-zero pixels 
which are surrounded by a perimeter of zero pixels in the 
position image. 
The above metiiods require relatively litUe computational 

35 power and memory to execute. Difference images are nor- 
mally available as a byproduct of interframe compression 
schemes. The difference image C-A may be generated by 
retaining difference image B-A and summing it with dif- 
ference image C-B. This saves space over the method of 

40 retaining the complete original image A for subtraction from 
image C since difference images contain many zero values 
which can be more corapacUy rep-csented than the original 
images. For purposes of moving object detection, the dif- 
ference images can be simplified by setting to zero pixel 

45 values that are close to zero. The purpose is to discard slight 
shadows and lighting shifts while retaining the sharp con- 
trast of the moving object itself. 

The image motion detection dius described can be used to 
direct the positioning of a high-resolution capnire window 

50 within an input image. In effect, the window can be designed 
to track moving objects to provide a more detailed image of 
persons or objects within the scene without drastically 
increasing the amount of image data to be stored. Useful 
windows which aid identification of a person within the field 

55 of view may typically rub from one-fourth to one-thirtieth of 
the image size. These windows are suited to intraframe 
compression (no difference frames) since they will likely 
change location within the image and their content is highly 
variable. At a typical 20:1 compression of the windowed 

60 contents, the windows generate a compressed data flow 
equivalent to full sized images coixqu^ssed at rates of $0:1 
to 600: 1. Thus, the full-time application of tracking win- 
dows is practical in terms of the data storage required, 

PRE-TRIGGER TECHNIQUES 

Digital data recording systems readily lend themselves to 
techniques for retaining data that arrives in advance of a 
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triggering event Consider for example a detector responsive 
to a door opening near an entryway and a camera fadng the 
same entryway. Receipt of a signal from the detector could 
trigger the recording system to retain camera images tba 
arrived prior to the opening of the door. In the case where 
camera images are normally captured at standard rates and 
resolutions, the trigger could cause a set of images to be 
retained at enhanced rates resolution. Thus a clearer or 
more frequent set of images of a person walking toward the 
door prior to opening the door wcxild later be available to aid 
in identification. These images are defined as pre-trigger 
images since they occur prior to the stimulus which causes 
them to be r^ned. Collection of enhanced images can 
continue for some interval after occurrence of the trigger 
event as well. Sudi images are defined as post-trigger 
images. Taken together, pre-trigger and post-trigger images 
provide a view, in the above example, of the person 
approaching, opening, and passing beyond the door. 

The central method of pre-trigger storage is to routinely 
retain sets of data representing enhanced image views in a 
storage buffer. This data is discarded a short time later if no 
trigger is received. When a trigger does arrive then pan or all 
of the temporary data is transferred to the long-term record 
as a *i7e-trigger' data sample. 

One method to retain pre-trigger images is to encode 
('encode' is synonymous with ^compress*) all source images 
at rates and resolutions specified for enhanced image views 
with a single encoder. The resultant compressed data is 
stored Into a memory buffer as shown in FIG. 35 A. A subset 
of the data in this memory buffer is normally written to mass 
storage such as to convey images at standard rates and 
resolution. A larger set of data is passed on to mass storage 
on receipt of a trigger in order to convey images at enhanced 
rates and resolution. The stcH'age buffer, which is preferably 
a solid-state memory, is located ahead of the mass storage 
device. The constant writing and clearing data inherent in 
pre-trigger recording methods is thus carried out on the 
storage buffer rather than on the mass storage device. A 
drawback with the approach of FIG. 35 A arises in the use of 
interframe image compression. A subset of images at lower 
rates and/cx* resolution than originally encoded cannot be 
readily extracted from a set of interframe con^ressed 
images. The most reliable method of extracting a reduced set 
of compressed images from an interframe compressed 
sequence requires deconq>ressing the sequence beginning at 
the key, (x first frame, and recon^ressing at the desired rate 
or resolution. 

FIG. 35B shows another arrangement for retaining pre- 
trigger images. Image encoder #2 and associated mass 
storage unit comprise a primary recording system without 
pre-trigger capture capability. Image encoder #1 and ^e 
storage buffer comprise a system for separately acquiring 
sets of data that can later be accessed by ttie primary 
recording system. 

In FIG. 35B. source images are acquired by image 
encoder #2 via path 2. Source images are also acquired by 
image encoder #1 via path 1, Images encoded by encoder #1 
are uransfeired to the storage buffer and normally retained 
for an interval equal to the pre-trigger recording duration, 
contents of the storage buffo* can be decoded by an image 
decoder and transfecred to mass storage by way of path 3 and 
image encoder #2, Path 3 is employed when it is desired to 
translate from the con^rcssed dau format of encoder #1 to 
the format of encoder #2. For example, were encoder #1 to 
generate an intraframe compressed format and encoder #2 
an interframe con^jressed format then translation would be 
readily accomplished by decompressing the intraframe data 
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from the. storage buffer and rccompressing to interframe 
format of image encoder #2. Alternately, contents of the 
stcH-age buffer can be transferred dfrectiy to mass storage via 
path 4 if the compression format is not to be altered. 
3 In the absence of triggers, all data to be retained in mass 
storage is normally encoded by image encoder #2 via path 

2. Enhanced images of the type that will be saved under 
trigger-initiated conditions are normally encoded by image 
encoder #1. It may be desired to retain enhanced images that 
occurred both prior to and subsequent to the trigger, that is, 
bodi pre-trigger and post-trigger images. Numerous strate- 
gies exist for accomplishing this. For example, post-trigger 
enhanced images can be encoded by image encoder #1 and 
these images can be included into the storage buffer along 
with the pre-trigger images collected earlier. Subsequently, 
all images in the storage buffer can be transferred to wain 
storage either directly by padi 4. or by re-encoding via path 

3. MothCT strategy is to freeze pre-trigger images in the 
storage buffer when a trigger arrives, and to acquire 
enhanced images as post-trigger images via image encoder 

^ #2. When post-trigger acquisition ceases, the pre-trigger 
images in the storage buffer can be re-encoded via image 
encoder #2. This strategy is illustrated in FIG. 36. 

In each diagram level A through E of FIG. 36. the 
rejffesentation of the contents of the storage buffer arc drawn 
above the representation of the contents of mass storage. The 
time scale advances to the right Standard image data is 
represented by short horizontally stacked blocks and 
enhanced data is represented by tall narrow stacked blocks. 
Levd A illustrates the conditions prior to trigger, where 
standard resolution data Is accumulated into mass storage 
and enhanced resolution data is temporarily retained in the 
storage buffer. The duration for which the data is retained in 
the storage buffa is suggested by the horizontal length of the 
sequence of storage buffer blocks. 

Level B depicts the arrival of a trigger event that will 
initiate the saving of pre-trigger and post-trigger image data. 
The trigger event causes contents of the storage buffer to be 
frozen as pre-trigger data; writing and clearing of the storage 

^ buffer is suspended. Level C dqjicts encoding of enhanced 
images by image encoder #2. These images are transferred 
into mass storage as post-trigger data. 

In Level the post-trigger encoding interval has ended. 
Now the process of re-encoding pre-trigger data contained in 

43 the storage buffer begins via the image decoder and image 
encoder #2 linked by path 3. Encoding of standard image 
data has resumed as well and so the re-encoding of pre- 
trigger data and encoding of standard data is done on a 
time-shared basis by image encoder #2, Blocks of pre- 
trigger data are unloaded from the storage buffer and 
re-encoded in the format of encoder #2. Finally, in Level E 
all pre-trigger data has been transferred to mass storage and 
image encoder #2 returns fully to the process of encoding 
standard image data. Image encoder #1 resumes encoding 

33 enhanced image data and the stor^e buffer refills. 

Note that inuiges are placed in mass storage out of time 
sequence; in particular, pre-trigger data is stored after post- 
trigger data. This does not represent a |X'oblem since each 
image is stored with a time stamp. When retrieved for 
viewing, the playback device uses the time stamp to replay 
the images in correct time-of-occurrence sequence. 

In one implementation of the dual encoder system of FIG. 
356. image encoder #1 is an intraframe encoder, such as 
J?EO, and image encoder #2 is an interframe encoder using 

65 key frames and difference frames. 

In another in^>lementation of the dual encoder system of 
FIG. 35B image encoder #1 compresses a high-resolution 
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window, or subset of the source image and image encoder 
#2 compresses the full image at standard resolution. Data 
from image encoder #1 is transferred to mass storage only 
under conditious of a trigger event The object of diis 
implementation is the selective, prc-triggcr and post-trigger 5 
recording of a high-resolution window of the input image. 

RAMIFICAnONS 

There oust a wide variety of fixed-media and removable- 
media storage devices that can be employed in implement- 
ing this invention. Others will undoubtedly be developed in 
die future. From a technical stanc^int the main criterion is 
ttiat the storage device be unable of storing and reproducing 
a significant quantity of digital data. Rresent-day removable- 
media storage devices that satisfy tfiis criterion include: 
digital magnetic tape drives, fl<^y disk drives, and 
re-writable optical disk drives. The primary fixed-media 
candidate for long-term storage is the nuignetic hard disk 
drive; integrated-drcuit memories are feasible for short- 
term storage or buffering. ^ 

Systems with combinations of storage devices having 
different features can be designed to optimize the perfor- 
mance for various applications. These features can be 
exploited both in the receding and archiving of data and in 
the retrieving and playing back of data. Some of these 
embodiments arc described below. 

A first embodiment of this invention uses a removable- 
media drive as die primary storage device. Digital t^ 
drives are a preferred primary storage device, though ^ 
re-writable optical disk drives, floppy disk drives, and other 
removable-media drives can be employed as well. The 
invention fills media cartridges at a user settable rate (e.g. 
once every 24 hours). The operator can maintain an ardiive 
of data by rotating media cartridges through a shelf storage 
for the duration of the archive interval. This method, shown 
in FIG. 3A, is practiced in conventional VCR surveillance 
installations. However, this invention provides tiie advan- 
tage of storing much more usable data in less f^ysical space. 

A disk drive in this first embodiment serves as a data ^ 
buffer ahead of the removable-media drive. This buffer 
allows the primary storage media to be removed for 
exchanging without loss of data. The buffer also allows 
writing the removable-media drive at rates that optimize life 
and reliability of the drive. Digital tapes have a significant 
advanUgc over VCRs in that the digital tape can generally 
write data much faster than the compressed images are 
generated. Thus, with buffering, the digitai tape need be 
moving only infrequently, and the drive can spend large 
fractions of time with rotating head and transport mecha- ^ 
oisms idle. 

A second embodiment of this invention utilizes a fixed- 
media drive, such as a magnetic hard-disk drive, as the 
primary storage device. Compressed images are retained on 
disk up to the limit of a usa-determined archive interval 55 
(e.g. 1 week), after which they are overwritten with new 
images. This method, shown in FIG. 3B, enables a system to 
record indefinitely without operator intervention, yet always 
have images from within the archive time interval available 
for playback. It is therefore useful for long-term unattended go 
recording in sites without dedicated surveillance systems 
personnel. 

FIG. 37 illustrates one possible application of a record/ 
playback device according to the fixed-media embodiment 
of the invention. This rccord/playback device will be 65 
referred to as an Unattended Recording Monitor 500. 
Record and playback functions for the unattended recording 
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monitor are represented in simplified form in FIG. 37. These 
functions have been described in detail elsewhere in this 
specification. 

Refeiring to FIG. 37. video signals from camera 1 502rl 
through camera 8 502-8 couple to video selector A 70 and 
video selector B 506. The output of video selector A is 
coupled to a record data i»ocessor 508, which includes the 
functions of video digitizer, image compressor, file manager, 
and rate control. The record data processor also accepts 
alarm data 504 from external sources. Data from the record 
data processor is delivered to a fixed-media stcrage sub- 
system 510 for storage. Data replacement processes 
described elsewhere in this specification remove data older 
than the archive interval to make room for newer data. 

Data for playback is received from the fixed-nffidia stor- 
age subsystem by a playback data processor 512. The 
playback data processor includes decompression and play- 
back time-stepping functions as described elsewhere in this 
specification. The output of the playback data processor 
couples to a video converter 142 and to a gr^^cs adaptor 
140. The video converter generates a composite or compo- 
nent video (e.g. NTSC or S- Video) output. The graphics 
ad^tor drives a graphics monitor 144. Output of the video 
converter can be routed through video selector B for viewing 
on monitor 66. 

In normal operation of the unattended recording monitor 
all video and alarm data is receded onto a fixed-media 
storage device contained in the fixed-media storage sub- 
system 510. Unlike cartridge media, the media in the storage 
sub-system cannot be removed and set aside when important 
data is to be preserved. Several means are provided therefore 
for ofQoading data of particular interest from the unattended 
recording monitor to other storage devices and to commu- 
nications diaimels. Conversely, data can be onloaded from 
these same storage devices and conuuunications channels to 
the unattended recording monitor. 

One form of a removable-raedia drive for offload storage 
is a flo{^y disk drive for reading and writing magnetic 
floppy disks. The floppy disk drive 520 connects to the 
fixed-media storage sub-system. Floppy disks are intended 
for transferring small portions of compressed digital data in 
and out of the unattended recording monitor. In most sur- 
veillance work an event of interest can be rq)resented with 
just a handful of images from a single camera view. If, for 
example, a five-gigabyte fixed-media drive is used to 
archive one week's worth of data from eight cameras, then 
a one-megabyte floppy disk can hold the equivalent of 16 
minutes of such video data from a single camera. This is 
adequate to contain most events of interest. Data can be 
offloaded to flc^y disks as a selective archive. Or data can 
be onloaded from floppy disks to the fixed-media storage 
sub-system for eventual playback review. 

Another form of removablo-media drive for offload/ 
onload storage is the digital tape drive for reading and 
writing digit^d magnetic tape cartridges. In the in^>lemen- 
tation of FIG. 37. a SCSI interface 522 connects to the 
fixed-media storage sub-system. The digital tape drive 524 
connects to this SCSI interface. Given the capacity of typical 
digital tape drives, it becomes practical to offload the entire 
contents of the fixed-storage media onto one or more digital 
tapes. 

A third form of removable-media offload storage is 
offered by a conventional analog VCR. The output of video 
converter 142 connects to the record input of a VCR 62, The 
playback output of this VCR connects to inputs of video 
selector A and video selector B. The unattended recording 
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monitor generates data for recording onto the VCR in one of 
two playback modes. In the first playback mode, images 
originating from a single video source are reproduced to 
match the rate at which tfiey were first recorded Repeat 
images may be inserted between active Image updates in the 5 
video output stream in order to maintain a correct time-of- 
occurrcocc in the playt>ack progression. This first playback 
mode would be used to view images on monitor 66 or to 
record onto a VCR tape for later direct viewing. 

A second playback mode generates data in a special ^0 
compact format The purpose of this compact fonnat is to 
transfa data to and from the VCR more rapidly than could 
be done under the first playback mode. Data recorded onto 
a VCR tape in compact format may not be directly viewable 
on a playback monitor. 15 

A number of contact formats are possible* One compact 
format removes all repeat francs from the video ou^ut 
stream. Repeat frames were used in the first playback mode 
to flu the time gaps between the originally recorded active 
image steps. Images originating from multiple video sources ^ 
can be interleaved using this format. Machine-readable 
codes are ttien insetted into the recorded video stream to 
later identify the video source associated witii each image. 

A second compact foroiat codes the compressed digital 
image data onto the analog video signal. This video signal 
can then be recorded onto and played back from a VCR In 
this way, the VCR serves as a digital offload/onload device 
for compressed video image data. The second con^ct 
format oflfers the twin advantages of digital coding, which 
prevents loss of image fidelity through VCR record and 
playback, and image compression, which permit a greater 
numbCT of images to be stored per tape. 

Compressed digital data is offloaded to a VCR by the 
following stq)s: Compressed data from the fixed-media 
storage sub-system 510 is first transfen^ed to the playback 
data processor 512. The playback data processor codes this 
data in a format suitable for impression onto an analog video 
signal The coded data is transferred to the video converter 
142 which converts the coded data to an analog video form ^ 
suitable for recording onto a VCR. This coded analog data 
is applied to the record input of VCR 62. 

Compressed digital data is onloaded from a VCR by the 
following steps: Coded analog video from die playback 
ou^ut of the VCR couples to video selector A 70. \^deo 45 
selector A routes this analog video to record data processor 
508 which digitizes the video and decodes the compact 
format The decoded data is transferred from the record data 
processor to the fixed-media storage sub-system. 

Offloading data to a VCR is a useful technique since one so 
videotape cartridge can hold a greater number of images 
than one floppy disk. In addition, VCRs are widely available 
in surveillance installations and in homes as consumer 
devices. The VCR thus serves as a means of archival cr 
backup storage for the unattended recording monitor. $5 

In addition to offload/onload storage devices, data can be 
offloaded to or onloaded from data networks such as local- 
area networks CLAN) or wide-area networks (WAN). A 
network interface 526 connects to the fixed-media Storage 
sub-system. The network interface connects to either a 60 
local-area network 528 or a wide-area network 530. 
Examples of local-area networks include Ethernet. Token- 
Ring and ATM (asynchronous-transfer mode). Examples of 
wide-area networks include telq>hone. ISDN, and Internet 

Network connections can be used to remotely poll the 65 
unattended recording monitor for alarm and other events. 
Specific image data can then be downloaded by way of the 
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network. Alto-nately. the unattended recording monitor can 
be progranuned to alert devices on the network in response 
to certain event criteria. These include loss of a camera 
signal, or alarm and building entry violations. Thirdly, the 
unattended recording monitor can be reprogranuned or 
re-configured by authorized devices on the network. 
Digita] implications 

Video cameras that ou^ut data directly in digital form are 
expected to proliferate. These cameras may transmit their 
data in either a parallel or serial fc^mat. Serial formats lend 
themselves to transmission over a fiber <^tic link. The 
recording device embodied in the invention can be config- 
ured to receive serial data via fiber optic inputs. 

FIG. 38 shows a portion of the record function for 
acquiring digital video ii^uts from tmilt^le asynchronous 
sources. This figure parallels the techniques for acquiring 
analog video described in connection with FIG. 8. In FIG, 38 
digital video streams 230A-D from video sources A through 
D connect to digital selector 234. The digital selector 
couples one of the video streams 230A-D to output 236 
under influence of selector control 114. The selector output 
carries an ou^t digital video stream containing images 
from the selected sources. Synchronization data 116 derived 
from the output digital video stream is received by acqui- 
sition controller 104. 

Acquisition of digital video by the system of FIG. 38 is 
similar to that described in connection with FIG. 8 and 
FIGS. 9A, 9B, and 9C. The acquisition controller causes the 
selected to remain connected to one video stream until 
digital data representing at least one whole field or frame has 
passed to the selector output. In this manner, video fields or 
flames can be switched at minimum rates of one-half the 
source stream rate. If the additional constraint of ordering 
the selection of streams according to the relative phases of 
the streams is applied then the field or frame switching rate 
increases to N/N+1 of the source rate, where N is the number 
of video sources. Phase registers 232A-D track the phase of 
video sources as described earlier in connection with FIG. 8 
and FIG. 9C. 

Although die present invention has been described using 
particular illustrative embodiments, it will be understood 
that many variations in construction, arrangement and use 
are possible within the scope of the invention. The present 
invention is thus to be considered as including all possible 
modifications and variations encompassed within the scope 
of the appended claims. 

What is daimed: 

1. A recording system for video image streams, said image 
streams having attributes including rate and resolution, said 
recording system coursing: 

an image processor, said image processor having an input 
for an image stream, an output for emitting a storage 
data stream, and at least one image attribute control 
input for varying one of said attributes of said st<H^e 
data stream; 

a storage sub-system, said storage sub-system accepting 
said storage data stream, wherein said storage sub- 
system permits overwriting of previously written data 
according to a replacement process, said storage sub- 
system having storage attributes including the rate of 
data entering storage, the volume of data accumulated 
in storage, the volume of storage space remaining, and 
a storage capacity, said storage capacity being equal to 
the sum of said volume of data accumulated in storage 
and said volume of storage space remaining; and 

a control means for dynamically adjusting the quantity of 
data entering said storage sub-system according to a 
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predetermined criteria by producing said at least one 
image attribute control input, said control means moni- 
toring at least one of said storage attributes, wherein 
said jH-edeiermined criteria during said replacement 
process is that said volume of data accumulated in s 
storage remains substantially equal to said storage 
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capacity and substantially all data younger ^an a 
progranmiable time interval are retained, said time 
interval being programmable without regard for said 
storage capacity. 

« * * * * 
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